Restricted Boltzmann Machines

Theory of RBMs and Applications

Introduction

Background

Restricted Boltzmann Machines (RBM) are a type of neural network that has been around since the 1980s. As a reminder to the reader, machine learning is generally divided into 3 categories: supervised learning (examples: classification tasks, regression), unsupervised learning (examples: clustering, dimensionality reduction, generative modeling), and reinforcement learning (examples: gaming/robotics). RBMs are primarily used for unsupervised learning tasks like dimensionality reduction and feature extraction, which help prepare datasets for machine learning models that may later be trained using supervised learning. They also have other applications which will be discussed further later.

Like Hopfield networks, Boltzmann machines are undirected graphical models, but they are different in that they are stochastic and can have hidden units. Both models are energy-based, meaning they learn by minimizing an energy function for each model (Smolensky et al. 1986). Boltzmann machines use a sigmoid activation function, which allows for the model to be probabilistic.

In the “Restricted” Boltzmann Machine, there are no interactions between neurons in the visible layer or between neurons in the hidden layer, creating a bipartite graph of neurons. Below is a diagram taken from Goodfellow, et al. (Goodfellow, Bengio, and Courville 2016) (p. 577) for visualization of the connections.

reticulate::py_config()

python:         /Users/jessicawells/.virtualenvs/r-reticulate/bin/python
libpython:      /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin/libpython3.12.dylib
pythonhome:     /Users/jessicawells/.virtualenvs/r-reticulate:/Users/jessicawells/.virtualenvs/r-reticulate
version:        3.12.6 (v3.12.6:a4a2d2b0d85, Sep  6 2024, 16:08:03) [Clang 13.0.0 (clang-1300.0.29.30)]
numpy:          /Users/jessicawells/.virtualenvs/r-reticulate/lib/python3.12/site-packages/numpy
numpy_version:  1.26.1

Goodfellow, et al. discuss the expense in drawing samples for most undirected graphical models; however, the RBM allows for block Gibbs sampling (p. 578) where the network alternates between sampling all hidden units simultaneously (etc. for visible). Derivatives are also simplified by the fact that the energy function of the RBM is a linear function of it’s parameters, which will be seen further in Methods.

RBMs are trained using a process called Contrastive Divergence (CD) (G. E. Hinton 2002) where the weights are updated to minimize the difference between samples from the data and samples from the model. Learning rate, batch size, and number of hidden units are all hyperparameters that can affect the ability of the training to converge successfully and learn the underlying structure of the data.

Applications

RBMs are probably best known for their success in collaborative filtering. The RBM model was used in the Netflix Prize competition to predict user ratings for movies, with the result that it outperformed the Singular Value Decomposition (SVD) method that was state-of-the-art at the time (Salakhutdinov, Mnih, and Hinton 2007). They have also been trained to recognize handwritten digits, such as the MNIST dataset (G. E. Hinton 2002).

RBMs have been successfully used to distinguish normal and anomalous network traffic. Their potential use in improving network security for companies in the future is promising. There is slow progress in network anomaly detection due to the difficulty of obtaining datasets for training and testing networks. Clients are often reluctant to divulge information that could potentially harm their networks. In a real-life dataset where one host had normal traffic and one was infected by a bot, discriminative RBM (DRBM) was able to successfully distinguish the normal from anomalous traffic. DRBM doesn’t rely on knowing the data distribution ahead of time, which is useful, except that it also causes the DRBM to overfit. As a result, when trying to use the same trained RBM on the KDD ’99 training dataset performance declined. (Fiore et al. 2013)

RBMs can provide greatly improved classification of brain disorders in MRI images. Generative Adversarial Networks (GANs) use two neural networks: a generator which generates fake data, and a discriminator which tries to distinguish between real and fake data. Loss from the discriminator is backpropagated through the generator so that both part are trained simultaneously. The RBM-GAN uses RBM features from real MRI images as inputs to the generator. Features from the discriminator are then used as inputs to a classifier. (Aslan, Dogan, and Koca 2023)

The many-body quantum wavefunction, which describes the quantum state of a system of particles is difficult to compute with classical computers. RBMs have been used to approximate it using variational Monte Carlo methods. (Melko et al. 2019)

RBMs are notoriously slow to train. The process of computing the activation probability requires the calculation of vector dot products. Lean Constrastive Divergence (LCD) is a method which adds two techniques to speed up the process of training RBMs. The first is bounds-based filtering where upper and lower bounds of the probability select only a range of dot products to perform. Second, the delta product involves only recalculating the changed portions of the vector dot product. (Ning, Pittman, and Shen 2018)

Methods

Below is the energy function of the RBM.

\[ E(v,h) = - \sum_{i} a_i v_i - \sum_{j} b_j h_j - \sum_{i} \sum_{j} v_i w_{i,j} h_j \tag{1}\] where v_i and h_j represent visible and hidden units; a_i and b_j are the bias terms of the visible and hidden units; and each w_{i,j} (weight) element represents the interaction between the visible and hidden units. (Fischer and Igel 2012)

It is well known neural networks are prone to overfitting and often techniques such as early stopping are employed to prevent it. Some methods to prevent overfitting in RBMs are weight decay (L2 regularization), dropout, dropconnect, and weight uncertainty (Zhang et al. 2018). Dropout is a fairly well known concept in deep learning. For example, a dropout value of 0.3 added to a layer means 30% of neurons are dropped during training. This prevents the network from learning certain features too well. L2 regularization is also a commonly employed technique in deep learning. It assigns a penalty to large weights to allow for more generalization. Dropconnect is a method where a subset of weights within the network are randomly set to zero during training. Weight uncertainty is where each weight in the network has it’s own probability distribution vice a fixed value. This addition allows the network to learn more useful features.

If the learning rate is too high, training of the model may not converge. If it is too low, training may take a long time. To fully maximize the training of the model it is helpful to reduce the learning rate over time. This is known as learning rate decay. (G. Hinton 2010)

Model Categories

We train Logistic Regression (with and without RBM features as input), Feed Forward Network (with and without RBM features as input), and Convolutional Neural Network. Below is a brief reminder of the basics of each model.

For the models incoroporating the RBM, we take the Fashion MNIST features/pixels and train the RBM (unsupervised learning) to extract hidden features from the visible layer and then feed these features into either logistic regression or feed forward network. We then use the trained model to predict labels for the test data, evaluating how well the RBM-derived features perform in a supervised classification task.

1. Logistic Regression

Mathematically, the concept behind binary logistic regression is the logit (the natural logarithm of an odds ratio)(Peng, Lee, and Ingersoll 2002). However, since we have 10 labels, our classification task falls into “Multinomial Logistic Regression.”

\[ P(Y = k | X) = \frac{e^{\beta_{0k} + \beta_k^T X}}{\sum_{l=1}^{K} e^{\beta_{0l} + \beta_l^T X}} \tag{2}\]

2. Simple Feed Forward Neural Network

The feed forward network (FNN) is one where information flows in one direction from input to output with no loops or feedback. There can be zero hidden layers in between (called single FNN) or one or more hidden layers (multilayer FNN).
(Sazlı 2006)

3. Convolutional Neural Network

The convolutional neural network (CNN) is a type of feed forward network except that unlike the traditional ANN, CNNs are primarily used for pattern recognition with images (O’Shea and Nash 2015). The CNN has 3 layers which are stacked to form the full CNN: convolutional, pooling, and fully-connected layers.

Below is our Process for creating the RBM:

Step 1: We first initialize the RBM with random weights and biases and set visible units to 784 and hidden units to 256. We also set the number of contrastive divergence steps (k) to 1.
Step 2: Sample hidden units from visible. The math behind computing the hidden unit activations from the given input can be seen in Equation 3 (Fischer and Igel 2012) where the probability is used to sample from the Bernoulli distribution.
\[ p(H_i = 1 | \mathbf{v}) = \sigma \left( \sum_{j=1}^{m} w_{ij} v_j + c_i \right) \tag{3}\]

where p(.) is the probability of the i^th hidden state being activated (=1) given the visible input vector. σ is the sigmoid activation function (below) which maps the weighted sum to a probability between 0 and 1. m is the number of visible units. w_ij is the weight connecting visible unit j to hidden unit i. v_j is the value of the j^th visible unit. and c_i is the bias term for the hidden unit. \[ \sigma(x) = \frac{1}{1 + e^{-x}} \]

Step 3: Sample visible units from hidden. The math behind computing visible unit activations from the hidden layer can be seen in Equation 4 (Fischer and Igel 2012) Visible states are sampled using the Bernoulli distribution. This way we can see how well the RBM learned from the inputs.
\[ p(V_j = 1 | \mathbf{h}) = \sigma \left( \sum_{i=1}^{n} w_{ij} h_i + b_j \right) \tag{4}\]

where p(.) is the probability of the i^th visible unit being activated (=1) given the hidden vector h. σ is same as above. n is the number of hidden units. w_ij is the weight connecting hidden unit i to visible unit j. b_j is the bias term for the j^th visible unit.
Step 4: K=1 steps of Contrastive Divergence (Feed Forward, Feed Backward) which executes steps 2 and 3. Contrastive Divergence updates the RBM’s weights by minimizing the difference between the original input and the reconstructed input created by the RBM.
Step 5: Free energy is computed. The free energy F is given by the logarithm of the partition function Z (Oh, Baggag, and Nha 2020) where the partition function is
\[ Z(\theta) \equiv \sum_{v,h} e^{-E(v,h; \theta)} \tag{5}\] and the free energy function is
\[ F(\theta) = -\ln Z(\theta) \tag{6}\] where lower free energy means the RBM learned the visible state well.

Step 6: Train the RBM. Model weights updated via gradient descent.
Step 7: Feature extraction for classification with LR. The hidden layer activations of the RBM are used as features for Logistic Regression and Feed Forward Network.

Hyperparameter Tuning

We use the Tree-structured Parzen Estimator algorithm from Optuna (Akiba et al. 2019) to tune the hyperparameters of the RBM and the classifier models, and we use MLFlow (Zaharia et al. 2018) to record and visualize the results of the hyperparameter tuning process. The hyperparameters we tune include the learning rate, batch size, number of hidden units, and number of epochs.

Metrics Used

1. Accuracy
Accuracy is defined as the number of correct classifications divided by the total number of classifications
\[ \text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}} \]

2. Macro F1 Score
Macro F1 score is the unweighted average of the individual F1 scores of each class. It takes no regard for class imbalance; however, we saw earlier the classes are all balanced in Fashion MNIST. The F1 score for each individual class is as follows \[ \text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \] where precision for each class is \[ \text{Precision} = \frac{TP}{TP + FP} \] and recall for each class is \[ \text{Recall} = \frac{TP}{TP + FN} \] The definitions of these terms for multiclass problems are more complicated than binary and are best displayed as examples.

Acronymn	Example for a trouser image
TP = True Positives	the image is a trouser and the model predicts a trouser
TN = True Negatives	the image is not a trouser and the model predicts anything but trouser
FP = False Positives	the image is anything but trouser but the model predicts trouser
FN = False Negatives	the image is a trouser and the model predicts another class (like shirt)

As stated earlier, the individual F1 scores for each class are taken and averaged to compute the Macro F1 score in a multiclass problem like Fashion MNIST.

Analysis and Results

Data Exploration and Visualization

We use the Fashion MNIST dataset from Zalando Research (Xiao, Rasul, and Vollgraf 2017). The set includes 70,000 grayscale images of clothing items, 60,000 for training and 10,000 for testing. Each image is 28x28 pixels (784 pixels total). Each pixel has a value associated with it ranging from 0 (white) to 255 (very dark) – whole numbers only. There are 785 columns in total as one column is dedicated to the label.

There are 10 labels in total:

0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot

Below we load the dataset.

import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
import torch
import torchvision.datasets
import torchvision.models
import torchvision.transforms as transforms
import matplotlib.pyplot as plt



train_data = torchvision.datasets.FashionMNIST(
    root="./data", 
    train=True, 
    download=True, 
    transform=transforms.ToTensor()  # Converts to tensor but does NOT normalize
)

test_data = torchvision.datasets.FashionMNIST(
    root="./data", 
    train=False, 
    download=True, 
    transform=transforms.ToTensor()  
)

Get the seventh image to show a sample

# Extract the first image (or choose any index)
image_tensor, label = train_data[6]  # shape: [1, 28, 28]

# Convert tensor to NumPy array
image_array = image_tensor.numpy().squeeze()  

# Plot the image
plt.figure(figsize=(5,5))
plt.imshow(image_array, cmap="gray")
plt.title(f"FashionMNIST Image (Label: {label})")
plt.axis("off")  # Hide axes

(-0.5, 27.5, 27.5, -0.5)

plt.show()

train_images = train_data.data.numpy()  # Raw pixel values (0-255)
train_labels = train_data.targets.numpy()
X = train_images.reshape(-1, 784)  # Flatten 28x28 images into 1D (60000, 784)

#print(train_images[:5])
flattened = train_images[:5].reshape(5, -1) 

# Create a DataFrame
df_flat = pd.DataFrame(flattened)
print(df_flat.head())

   0    1    2    3    4    5    6    ...  777  778  779  780  781  782  783
0    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0
1    0    0    0    0    0    1    0  ...   76    0    0    0    0    0    0
2    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0
3    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0
4    0    0    0    0    0    0    0  ...    0    0    0    0    0    0    0

[5 rows x 784 columns]

#train_df.info() #datatypes are integers

There are no missing values in the data.

print(np.isnan(train_images).any()) 

False

There appears to be no class imbalance

unique_labels, counts = np.unique(train_labels, return_counts=True)

# Print the counts sorted by label
for label, count in zip(unique_labels, counts):
    print(f"Label {label}: {count}")

Label 0: 6000
Label 1: 6000
Label 2: 6000
Label 3: 6000
Label 4: 6000
Label 5: 6000
Label 6: 6000
Label 7: 6000
Label 8: 6000
Label 9: 6000

print(f"X shape: {X.shape}")

X shape: (60000, 784)

t-SNE Visualization
t-distributed Stochastic Neighbor Embedding (t-SNE) is used here to visualize the separation between classes in a high-dimensional dataset.
Each point represents a single fashion item (e.g., T-shirt, Trouser, etc.), and the color corresponds to its true label across the 10 categories listed above.

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

# Run t-SNE to reduce dimensionality
#embeddings = TSNE(n_jobs=2).fit_transform(X)

tsne = TSNE(n_jobs=-1, random_state=42)  # Use -1 to use all available cores
embeddings = tsne.fit_transform(X) #use scikitlearn instead


# Create scatter plot
figure = plt.figure(figsize=(15,7))
plt.scatter(embeddings[:, 0], embeddings[:, 1], c=train_labels,
            cmap=plt.cm.get_cmap("jet", 10), marker='.')
plt.colorbar(ticks=range(10))

<matplotlib.colorbar.Colorbar object at 0x3194f5460>

plt.clim(-0.5, 9.5)
plt.title("t-SNE Visualization of Fashion MNIST")
plt.show()

What the visualization shows:
Class 1 (blue / Trousers) forms a clearly distinct and tightly packed cluster, indicating that the pixel patterns for trousers are less similar to those of other classes. In contrast, Classes 4 (Coat), 6 (Shirt), and 2 (Pullover) show significant overlap, suggesting that these clothing items are harder to distinguish visually and may lead to more confusion during classification.

Modeling and Results

Our Goal
We are classifying Fashion MNIST images into one of 10 categories. To evaluate performance, we’re comparing five different models — some trained on raw pixel values and others using features extracted by a Restricted Boltzmann Machine (RBM). Our objective is to assess whether incorporating RBM into the workflow improves classification accuracy compared to using raw image data alone.

Our Models
1. Logistic Regression on Fashion MNIST Data
2. Feed Forward Network on Fashion MNIST Data
3. Convolutional Neural Network on Fashion MNIST Data
4. Logistic Regression on RBM Hidden Features (of Fashion MNIST Data)
5. Feed Forward Network on RBM Hidden Features (of Fashion MNIST Data)

Note: Outputs (50 trials) and Code are below for each model. Both the code and output can be toggled by the reader.
• The first click reveals a toggle labeled “Code”.
• Clicking “Code” will show the output.
• Clicking again will switch from output to the actual code.
• Clicking “Show Code and Output” again will collapse both views.

Import Libraries and Re-load data for first 3 models

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torchvision import datasets, transforms
import numpy as np
import mlflow
import optuna
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from torch.utils.data import DataLoader

# Set device
device = torch.device("mps")

# Load Fashion-MNIST dataset again for the first 3 models
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.FashionMNIST(root='./data', train=True, transform=transform, download=True)
test_dataset = datasets.FashionMNIST(root='./data', train=False, transform=transform, download=True)

#mlflow.end_run()
#run this in terminal when need to fully clean out expierment after you delete it in the ui
#rm -rf mlruns/.trash/*

Model 1: Logistic Regression on Fashion MNIST Data

Click to Show Code and Output

from sklearn.metrics import f1_score

CLASSIFIER = "LogisticRegression"  # Change for FNN, LogisticRegression, or CNN



# Define CNN model
class FashionCNN(nn.Module):
    def __init__(self, filters1, filters2, kernel1, kernel2):
        super(FashionCNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=filters1, kernel_size=kernel1, padding=1),
            nn.BatchNorm2d(filters1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=filters1, out_channels=filters2, kernel_size=kernel2),
            nn.BatchNorm2d(filters2),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        self.fc1 = None
        self.drop = nn.Dropout2d(0.25)
        self.fc2 = nn.Linear(in_features=600, out_features=120)
        self.fc3 = nn.Linear(in_features=120, out_features=10)
        

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        if self.fc1 is None:
            self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)
        out = self.fc1(out)
        out = self.drop(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out

        # Dynamically calculate flattened size
        out = out.view(out.size(0), -1)  # Flatten
        if self.fc1 is None:
            self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)  # ✅ Update FC layer dynamically

        out = self.fc1(out)
        out = self.drop(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out




# Define Optuna objective function
def objective(trial):
      # Set MLflow experiment name
    if CLASSIFIER == "LogisticRegression":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-noRBM")
    elif CLASSIFIER == "FNN":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-noRBM")
    elif CLASSIFIER == "CNN":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-cnn-noRBM")
    batch_size = trial.suggest_int("batch_size", 64, 256, step=32)
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

    mlflow.start_run(experiment_id=experiment.experiment_id)
    num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5) 
    mlflow.log_param("num_classifier_epochs", num_classifier_epochs)

    if CLASSIFIER == "FNN":
        hidden_size = trial.suggest_int("fnn_hidden", 192, 384)
        learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)

        mlflow.log_param("classifier", "FNN")
        mlflow.log_param("fnn_hidden", hidden_size)
        mlflow.log_param("learning_rate", learning_rate)

        model = nn.Sequential(
            nn.Linear(784, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, 10)
        ).to(device)

        optimizer = optim.Adam(model.parameters(), lr=learning_rate)

    elif CLASSIFIER == "CNN":
        filters1 = trial.suggest_int("filters1", 16, 64, step=16)
        filters2 = trial.suggest_int("filters2", 32, 128, step=32)
        kernel1 = trial.suggest_int("kernel1", 3, 5)
        kernel2 = trial.suggest_int("kernel2", 3, 5)
        learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)

        mlflow.log_param("classifier", "CNN")
        mlflow.log_param("filters1", filters1)
        mlflow.log_param("filters2", filters2)
        mlflow.log_param("kernel1", kernel1)
        mlflow.log_param("kernel2", kernel2)
        mlflow.log_param("learning_rate", learning_rate)

        model = FashionCNN(filters1, filters2, kernel1, kernel2).to(device)
        optimizer = optim.Adam(model.parameters(), lr=learning_rate)

      
    elif CLASSIFIER == "LogisticRegression":
        mlflow.log_param("classifier", "LogisticRegression")
    
        # Prepare data for Logistic Regression (Flatten 28x28 images to 784 features)
        train_features = train_dataset.data.view(-1, 784).numpy()
        train_labels = train_dataset.targets.numpy()
        test_features = test_dataset.data.view(-1, 784).numpy()
        test_labels = test_dataset.targets.numpy()
    
        # Normalize the pixel values to [0,1] for better convergence
        train_features = train_features / 255.0
        test_features = test_features / 255.0
    
    
        C = trial.suggest_float("C", 0.01, 10.0, log=True)  
        solver = "saga" 
    
        model = LogisticRegression(C=C, max_iter=num_classifier_epochs, solver=solver)
        model.fit(train_features, train_labels)
    
    
        predictions = model.predict(test_features)
        accuracy = accuracy_score(test_labels, predictions) * 100
        
        macro_f1 = f1_score(test_labels, predictions, average="macro") #for f1
        print(f"Logistic Regression Test Accuracy: {accuracy:.2f}%")
        print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
    
        mlflow.log_param("C", C)
        mlflow.log_metric("test_accuracy", accuracy)
        mlflow.log_metric("macro_f1", macro_f1) #for f1
        mlflow.end_run()
        return accuracy

    # Training Loop for FNN and CNN
    criterion = nn.CrossEntropyLoss()

    model.train()
    for epoch in range(num_classifier_epochs):
        running_loss = 0.0
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))

            optimizer.zero_grad()
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        print(f"{CLASSIFIER} Epoch {epoch+1}: loss = {running_loss / len(train_loader):.4f}")

    # Model Evaluation
    model.eval()
    correct, total = 0, 0
    all_preds = []   # for f1
    all_labels = [] 
    with torch.no_grad():
        for images, labels in test_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            all_preds.extend(predicted.cpu().numpy())   #for f1
            all_labels.extend(labels.cpu().numpy()) #for f1

    accuracy = 100 * correct / total
    macro_f1 = f1_score(all_labels, all_preds, average="macro") #for f1
    print(f"Test Accuracy: {accuracy:.2f}%")
    print(f"Macro F1 Score: {macro_f1:.4f}") #for f1

    mlflow.log_metric("test_accuracy", accuracy)
    mlflow.log_metric("macro_f1", macro_f1) #for f1
    mlflow.end_run()
    return accuracy

if __name__ == "__main__":
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=50)
    print(f"Best Parameters for {CLASSIFIER}:", study.best_params)
    print("Best Accuracy:", study.best_value)

Logistic Regression Test Accuracy: 84.64%
Macro F1 Score: 0.8458
Logistic Regression Test Accuracy: 84.29%
Macro F1 Score: 0.8415
Logistic Regression Test Accuracy: 84.60%
Macro F1 Score: 0.8452
Logistic Regression Test Accuracy: 84.20%
Macro F1 Score: 0.8412
Logistic Regression Test Accuracy: 84.47%
Macro F1 Score: 0.8438
Logistic Regression Test Accuracy: 84.15%
Macro F1 Score: 0.8403
Logistic Regression Test Accuracy: 84.45%
Macro F1 Score: 0.8442
Logistic Regression Test Accuracy: 84.42%
Macro F1 Score: 0.8432
Logistic Regression Test Accuracy: 84.41%
Macro F1 Score: 0.8431
Logistic Regression Test Accuracy: 84.52%
Macro F1 Score: 0.8445
Logistic Regression Test Accuracy: 84.48%
Macro F1 Score: 0.8439
Logistic Regression Test Accuracy: 84.61%
Macro F1 Score: 0.8449
Logistic Regression Test Accuracy: 84.54%
Macro F1 Score: 0.8444
Logistic Regression Test Accuracy: 84.41%
Macro F1 Score: 0.8435
Logistic Regression Test Accuracy: 84.52%
Macro F1 Score: 0.8447
Logistic Regression Test Accuracy: 84.45%
Macro F1 Score: 0.8438
Logistic Regression Test Accuracy: 84.47%
Macro F1 Score: 0.8434
Logistic Regression Test Accuracy: 84.52%
Macro F1 Score: 0.8445
Logistic Regression Test Accuracy: 84.42%
Macro F1 Score: 0.8437
Logistic Regression Test Accuracy: 84.51%
Macro F1 Score: 0.8444
Logistic Regression Test Accuracy: 84.51%
Macro F1 Score: 0.8434
Logistic Regression Test Accuracy: 84.48%
Macro F1 Score: 0.8440
Logistic Regression Test Accuracy: 84.51%
Macro F1 Score: 0.8438
Logistic Regression Test Accuracy: 84.59%
Macro F1 Score: 0.8447
Logistic Regression Test Accuracy: 84.67%
Macro F1 Score: 0.8458
Logistic Regression Test Accuracy: 84.40%
Macro F1 Score: 0.8440
Logistic Regression Test Accuracy: 84.67%
Macro F1 Score: 0.8458
Logistic Regression Test Accuracy: 84.50%
Macro F1 Score: 0.8439
Logistic Regression Test Accuracy: 84.47%
Macro F1 Score: 0.8438
Logistic Regression Test Accuracy: 84.52%
Macro F1 Score: 0.8440
Logistic Regression Test Accuracy: 84.56%
Macro F1 Score: 0.8449
Logistic Regression Test Accuracy: 84.62%
Macro F1 Score: 0.8452
Logistic Regression Test Accuracy: 84.55%
Macro F1 Score: 0.8441
Logistic Regression Test Accuracy: 84.54%
Macro F1 Score: 0.8442
Logistic Regression Test Accuracy: 84.48%
Macro F1 Score: 0.8441
Logistic Regression Test Accuracy: 84.58%
Macro F1 Score: 0.8450
Logistic Regression Test Accuracy: 84.44%
Macro F1 Score: 0.8434
Logistic Regression Test Accuracy: 84.61%
Macro F1 Score: 0.8452
Logistic Regression Test Accuracy: 84.54%
Macro F1 Score: 0.8450
Logistic Regression Test Accuracy: 84.54%
Macro F1 Score: 0.8441
Logistic Regression Test Accuracy: 84.49%
Macro F1 Score: 0.8443
Logistic Regression Test Accuracy: 84.60%
Macro F1 Score: 0.8453
Logistic Regression Test Accuracy: 84.51%
Macro F1 Score: 0.8443
Logistic Regression Test Accuracy: 84.62%
Macro F1 Score: 0.8453
Logistic Regression Test Accuracy: 84.52%
Macro F1 Score: 0.8444
Logistic Regression Test Accuracy: 84.57%
Macro F1 Score: 0.8446
Logistic Regression Test Accuracy: 84.48%
Macro F1 Score: 0.8444
Logistic Regression Test Accuracy: 84.43%
Macro F1 Score: 0.8436
Logistic Regression Test Accuracy: 84.22%
Macro F1 Score: 0.8407
Logistic Regression Test Accuracy: 84.43%
Macro F1 Score: 0.8439
Best Parameters for LogisticRegression: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.9910546414812195}
Best Accuracy: 84.67

[I 2025-03-28 14:03:17,300] A new study created in memory with name: no-name-a763725a-0f38-4792-8ede-a3a5148e6dbc
[I 2025-03-28 14:03:24,080] Trial 0 finished with value: 84.64 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 4.005755197570405}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:03:30,796] Trial 1 finished with value: 84.28999999999999 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 0.029468017102199123}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:03:37,401] Trial 2 finished with value: 84.6 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.6212506932865256}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:03:44,073] Trial 3 finished with value: 84.2 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 0.020955917529814606}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:03:50,782] Trial 4 finished with value: 84.47 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'C': 0.3121125440383706}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:03:57,481] Trial 5 finished with value: 84.15 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'C': 0.012386979715403923}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:04,158] Trial 6 finished with value: 84.45 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'C': 0.19973940966972406}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:10,859] Trial 7 finished with value: 84.42 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 0.05330585940873715}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:17,548] Trial 8 finished with value: 84.41 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'C': 0.09050694362352474}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:24,157] Trial 9 finished with value: 84.52 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 0.05230937969970407}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:30,816] Trial 10 finished with value: 84.48 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'C': 7.3972194801993325}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:37,468] Trial 11 finished with value: 84.61 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 2.5511072593183397}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:44,180] Trial 12 finished with value: 84.54 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 3.853822640057023}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:50,803] Trial 13 finished with value: 84.41 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 1.657072564650452}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:04:57,541] Trial 14 finished with value: 84.52 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'C': 1.5754830120834762}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:04,206] Trial 15 finished with value: 84.45 and parameters: {'batch_size': 256, 'num_classifier_epochs': 5, 'C': 9.303014569590758}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:10,862] Trial 16 finished with value: 84.47 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 1.9593403650111556}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:17,513] Trial 17 finished with value: 84.52 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.756116552630944}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:24,185] Trial 18 finished with value: 84.42 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 4.021504777647571}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:30,751] Trial 19 finished with value: 84.50999999999999 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'C': 0.7938897482060593}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:37,318] Trial 20 finished with value: 84.50999999999999 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'C': 4.117376299327255}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:44,003] Trial 21 finished with value: 84.48 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.3828085370182236}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:50,572] Trial 22 finished with value: 84.50999999999999 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.8485932090577519}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:05:57,182] Trial 23 finished with value: 84.59 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 2.5624426088303056}. Best is trial 0 with value: 84.64.
[I 2025-03-28 14:06:03,715] Trial 24 finished with value: 84.67 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.9910546414812195}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:10,248] Trial 25 finished with value: 84.39999999999999 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 1.2655095430293073}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:16,856] Trial 26 finished with value: 84.67 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 5.8825032994772455}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:23,390] Trial 27 finished with value: 84.5 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'C': 6.088761728305264}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:30,022] Trial 28 finished with value: 84.47 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 5.311149351588817}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:36,565] Trial 29 finished with value: 84.52 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 9.85585339156705}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:43,167] Trial 30 finished with value: 84.56 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'C': 3.038695101823179}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:49,784] Trial 31 finished with value: 84.61999999999999 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 2.550424666091026}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:06:56,403] Trial 32 finished with value: 84.55 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 1.2651919043301858}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:02,980] Trial 33 finished with value: 84.54 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.43525367961019507}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:09,643] Trial 34 finished with value: 84.48 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 5.306178723559651}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:16,322] Trial 35 finished with value: 84.58 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 2.515204699890743}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:22,938] Trial 36 finished with value: 84.44 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.19782092093669573}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:29,476] Trial 37 finished with value: 84.61 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'C': 0.5468215159366026}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:36,026] Trial 38 finished with value: 84.54 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 0.2167562124440949}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:42,635] Trial 39 finished with value: 84.54 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 1.0824430211479827}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:49,245] Trial 40 finished with value: 84.49 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 3.2944308410829497}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:07:55,859] Trial 41 finished with value: 84.6 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 2.104154517495734}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:02,484] Trial 42 finished with value: 84.50999999999999 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'C': 6.420389154315065}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:09,130] Trial 43 finished with value: 84.61999999999999 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 4.412163667982693}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:15,750] Trial 44 finished with value: 84.52 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 4.663741408256952}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:22,338] Trial 45 finished with value: 84.57000000000001 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 7.880736706498457}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:28,992] Trial 46 finished with value: 84.48 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 1.8031331902833885}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:35,668] Trial 47 finished with value: 84.43 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 3.1167541278023543}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:42,345] Trial 48 finished with value: 84.22 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'C': 0.02127464220882997}. Best is trial 24 with value: 84.67.
[I 2025-03-28 14:08:48,965] Trial 49 finished with value: 84.43 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'C': 7.184870291367165}. Best is trial 24 with value: 84.67.

Test Accuracy of Logistic Regression by C (inverse regularization strength)

Model 2: Feed Forward Network on Fashion MNIST Data

Click to Show Code and Output

from sklearn.metrics import f1_score

CLASSIFIER = "FNN"  # Change for FNN, LogisticRegression, or CNN

# Define CNN model
class FashionCNN(nn.Module):
    def __init__(self, filters1, filters2, kernel1, kernel2):
        super(FashionCNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=filters1, kernel_size=kernel1, padding=1),
            nn.BatchNorm2d(filters1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=filters1, out_channels=filters2, kernel_size=kernel2),
            nn.BatchNorm2d(filters2),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        self.fc1 = None
        self.drop = nn.Dropout2d(0.25)
        self.fc2 = nn.Linear(in_features=600, out_features=120)
        self.fc3 = nn.Linear(in_features=120, out_features=10)
        

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        if self.fc1 is None:
            self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)
        out = self.fc1(out)
        out = self.drop(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out

        # Dynamically calculate flattened size
        out = out.view(out.size(0), -1)  # Flatten
        if self.fc1 is None:
            self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)  # ✅ Update FC layer dynamically

        out = self.fc1(out)
        out = self.drop(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out




# Define Optuna objective function
def objective(trial):
      # Set MLflow experiment name
    if CLASSIFIER == "LogisticRegression":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-noRBM")
    elif CLASSIFIER == "FNN":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-noRBM")
    elif CLASSIFIER == "CNN":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-cnn-noRBM")
    batch_size = trial.suggest_int("batch_size", 64, 256, step=32)
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

    mlflow.start_run(experiment_id=experiment.experiment_id)
    num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5) 
    mlflow.log_param("num_classifier_epochs", num_classifier_epochs)

    if CLASSIFIER == "FNN":
        hidden_size = trial.suggest_int("fnn_hidden", 192, 384)
        learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)

        mlflow.log_param("classifier", "FNN")
        mlflow.log_param("fnn_hidden", hidden_size)
        mlflow.log_param("learning_rate", learning_rate)

        model = nn.Sequential(
            nn.Linear(784, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, 10)
        ).to(device)

        optimizer = optim.Adam(model.parameters(), lr=learning_rate)

    elif CLASSIFIER == "CNN":
        filters1 = trial.suggest_int("filters1", 16, 64, step=16)
        filters2 = trial.suggest_int("filters2", 32, 128, step=32)
        kernel1 = trial.suggest_int("kernel1", 3, 5)
        kernel2 = trial.suggest_int("kernel2", 3, 5)
        learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)

        mlflow.log_param("classifier", "CNN")
        mlflow.log_param("filters1", filters1)
        mlflow.log_param("filters2", filters2)
        mlflow.log_param("kernel1", kernel1)
        mlflow.log_param("kernel2", kernel2)
        mlflow.log_param("learning_rate", learning_rate)

        model = FashionCNN(filters1, filters2, kernel1, kernel2).to(device)
        optimizer = optim.Adam(model.parameters(), lr=learning_rate)

      
    elif CLASSIFIER == "LogisticRegression":
        mlflow.log_param("classifier", "LogisticRegression")
    
        # Prepare data for Logistic Regression (Flatten 28x28 images to 784 features)
        train_features = train_dataset.data.view(-1, 784).numpy()
        train_labels = train_dataset.targets.numpy()
        test_features = test_dataset.data.view(-1, 784).numpy()
        test_labels = test_dataset.targets.numpy()
    
        # Normalize the pixel values to [0,1] for better convergence
        train_features = train_features / 255.0
        test_features = test_features / 255.0
    
    
        C = trial.suggest_float("C", 0.01, 10.0, log=True)  
        solver = "saga" 
    
        model = LogisticRegression(C=C, max_iter=num_classifier_epochs, solver=solver)
        model.fit(train_features, train_labels)
    
    
        predictions = model.predict(test_features)
        accuracy = accuracy_score(test_labels, predictions) * 100
        
        macro_f1 = f1_score(test_labels, predictions, average="macro") #for f1
        print(f"Logistic Regression Test Accuracy: {accuracy:.2f}%")
        print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
    
        mlflow.log_param("C", C)
        mlflow.log_metric("test_accuracy", accuracy)
        mlflow.log_metric("macro_f1", macro_f1) #for f1
        mlflow.end_run()
        return accuracy

    # Training Loop for FNN and CNN
    criterion = nn.CrossEntropyLoss()

    model.train()
    for epoch in range(num_classifier_epochs):
        running_loss = 0.0
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))

            optimizer.zero_grad()
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        print(f"{CLASSIFIER} Epoch {epoch+1}: loss = {running_loss / len(train_loader):.4f}")

    # Model Evaluation
    model.eval()
    correct, total = 0, 0
    all_preds = []   # for f1
    all_labels = [] 
    with torch.no_grad():
        for images, labels in test_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            all_preds.extend(predicted.cpu().numpy())   #for f1
            all_labels.extend(labels.cpu().numpy()) #for f1

    accuracy = 100 * correct / total
    macro_f1 = f1_score(all_labels, all_preds, average="macro") #for f1
    print(f"Test Accuracy: {accuracy:.2f}%")
    print(f"Macro F1 Score: {macro_f1:.4f}") #for f1

    mlflow.log_metric("test_accuracy", accuracy)
    mlflow.log_metric("macro_f1", macro_f1) #for f1
    mlflow.end_run()
    return accuracy

if __name__ == "__main__":
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=50)
    print(f"Best Parameters for {CLASSIFIER}:", study.best_params)
    print("Best Accuracy:", study.best_value)

FNN Epoch 1: loss = 0.5087
FNN Epoch 2: loss = 0.3764
FNN Epoch 3: loss = 0.3368
FNN Epoch 4: loss = 0.3098
FNN Epoch 5: loss = 0.2934
Test Accuracy: 87.28%
Macro F1 Score: 0.8673
FNN Epoch 1: loss = 0.6897
FNN Epoch 2: loss = 0.4514
FNN Epoch 3: loss = 0.4053
FNN Epoch 4: loss = 0.3766
FNN Epoch 5: loss = 0.3588
Test Accuracy: 85.83%
Macro F1 Score: 0.8543
FNN Epoch 1: loss = 0.5476
FNN Epoch 2: loss = 0.3894
FNN Epoch 3: loss = 0.3446
FNN Epoch 4: loss = 0.3158
FNN Epoch 5: loss = 0.2967
Test Accuracy: 87.59%
Macro F1 Score: 0.8761
FNN Epoch 1: loss = 0.6250
FNN Epoch 2: loss = 0.4280
FNN Epoch 3: loss = 0.3871
FNN Epoch 4: loss = 0.3577
FNN Epoch 5: loss = 0.3362
Test Accuracy: 86.81%
Macro F1 Score: 0.8653
FNN Epoch 1: loss = 0.5223
FNN Epoch 2: loss = 0.3784
FNN Epoch 3: loss = 0.3355
FNN Epoch 4: loss = 0.3142
FNN Epoch 5: loss = 0.2912
Test Accuracy: 87.62%
Macro F1 Score: 0.8757
FNN Epoch 1: loss = 0.5931
FNN Epoch 2: loss = 0.4156
FNN Epoch 3: loss = 0.3729
FNN Epoch 4: loss = 0.3451
FNN Epoch 5: loss = 0.3290
Test Accuracy: 86.71%
Macro F1 Score: 0.8650
FNN Epoch 1: loss = 0.5084
FNN Epoch 2: loss = 0.3782
FNN Epoch 3: loss = 0.3366
FNN Epoch 4: loss = 0.3124
FNN Epoch 5: loss = 0.2921
Test Accuracy: 87.95%
Macro F1 Score: 0.8784
FNN Epoch 1: loss = 1.0307
FNN Epoch 2: loss = 0.5889
FNN Epoch 3: loss = 0.5123
FNN Epoch 4: loss = 0.4749
FNN Epoch 5: loss = 0.4493
Test Accuracy: 83.61%
Macro F1 Score: 0.8337
FNN Epoch 1: loss = 0.5636
FNN Epoch 2: loss = 0.3984
FNN Epoch 3: loss = 0.3531
FNN Epoch 4: loss = 0.3285
FNN Epoch 5: loss = 0.3064
Test Accuracy: 86.77%
Macro F1 Score: 0.8655
FNN Epoch 1: loss = 0.8425
FNN Epoch 2: loss = 0.5113
FNN Epoch 3: loss = 0.4562
FNN Epoch 4: loss = 0.4261
FNN Epoch 5: loss = 0.4051
Test Accuracy: 85.08%
Macro F1 Score: 0.8500
FNN Epoch 1: loss = 0.5062
FNN Epoch 2: loss = 0.3709
FNN Epoch 3: loss = 0.3365
FNN Epoch 4: loss = 0.3101
FNN Epoch 5: loss = 0.2914
Test Accuracy: 86.99%
Macro F1 Score: 0.8688
FNN Epoch 1: loss = 0.4901
FNN Epoch 2: loss = 0.3667
FNN Epoch 3: loss = 0.3305
FNN Epoch 4: loss = 0.3047
FNN Epoch 5: loss = 0.2863
Test Accuracy: 87.77%
Macro F1 Score: 0.8780
FNN Epoch 1: loss = 0.4942
FNN Epoch 2: loss = 0.3671
FNN Epoch 3: loss = 0.3286
FNN Epoch 4: loss = 0.3050
FNN Epoch 5: loss = 0.2882
Test Accuracy: 87.48%
Macro F1 Score: 0.8760
FNN Epoch 1: loss = 0.5249
FNN Epoch 2: loss = 0.3827
FNN Epoch 3: loss = 0.3405
FNN Epoch 4: loss = 0.3160
FNN Epoch 5: loss = 0.2958
Test Accuracy: 85.96%
Macro F1 Score: 0.8622
FNN Epoch 1: loss = 0.5469
FNN Epoch 2: loss = 0.3992
FNN Epoch 3: loss = 0.3564
FNN Epoch 4: loss = 0.3279
FNN Epoch 5: loss = 0.3082
Test Accuracy: 87.03%
Macro F1 Score: 0.8688
FNN Epoch 1: loss = 0.5054
FNN Epoch 2: loss = 0.3690
FNN Epoch 3: loss = 0.3302
FNN Epoch 4: loss = 0.3059
FNN Epoch 5: loss = 0.2908
Test Accuracy: 87.53%
Macro F1 Score: 0.8738
FNN Epoch 1: loss = 0.5027
FNN Epoch 2: loss = 0.3665
FNN Epoch 3: loss = 0.3294
FNN Epoch 4: loss = 0.3087
FNN Epoch 5: loss = 0.2909
Test Accuracy: 87.50%
Macro F1 Score: 0.8738
FNN Epoch 1: loss = 0.5466
FNN Epoch 2: loss = 0.3952
FNN Epoch 3: loss = 0.3516
FNN Epoch 4: loss = 0.3246
FNN Epoch 5: loss = 0.2997
Test Accuracy: 87.29%
Macro F1 Score: 0.8709
FNN Epoch 1: loss = 0.6288
FNN Epoch 2: loss = 0.4358
FNN Epoch 3: loss = 0.3942
FNN Epoch 4: loss = 0.3681
FNN Epoch 5: loss = 0.3479
Test Accuracy: 86.39%
Macro F1 Score: 0.8649
FNN Epoch 1: loss = 0.5035
FNN Epoch 2: loss = 0.3697
FNN Epoch 3: loss = 0.3303
FNN Epoch 4: loss = 0.3064
FNN Epoch 5: loss = 0.2906
Test Accuracy: 87.66%
Macro F1 Score: 0.8754
FNN Epoch 1: loss = 0.4961
FNN Epoch 2: loss = 0.3652
FNN Epoch 3: loss = 0.3309
FNN Epoch 4: loss = 0.3078
FNN Epoch 5: loss = 0.2920
Test Accuracy: 86.23%
Macro F1 Score: 0.8587
FNN Epoch 1: loss = 0.4997
FNN Epoch 2: loss = 0.3696
FNN Epoch 3: loss = 0.3328
FNN Epoch 4: loss = 0.3061
FNN Epoch 5: loss = 0.2899
Test Accuracy: 87.66%
Macro F1 Score: 0.8746
FNN Epoch 1: loss = 0.4945
FNN Epoch 2: loss = 0.3645
FNN Epoch 3: loss = 0.3292
FNN Epoch 4: loss = 0.3084
FNN Epoch 5: loss = 0.2878
Test Accuracy: 87.65%
Macro F1 Score: 0.8757
FNN Epoch 1: loss = 0.5335
FNN Epoch 2: loss = 0.3910
FNN Epoch 3: loss = 0.3472
FNN Epoch 4: loss = 0.3213
FNN Epoch 5: loss = 0.3000
Test Accuracy: 87.11%
Macro F1 Score: 0.8709
FNN Epoch 1: loss = 0.5038
FNN Epoch 2: loss = 0.3696
FNN Epoch 3: loss = 0.3325
FNN Epoch 4: loss = 0.3082
FNN Epoch 5: loss = 0.2882
Test Accuracy: 86.11%
Macro F1 Score: 0.8625
FNN Epoch 1: loss = 0.5646
FNN Epoch 2: loss = 0.3979
FNN Epoch 3: loss = 0.3554
FNN Epoch 4: loss = 0.3274
FNN Epoch 5: loss = 0.3064
Test Accuracy: 87.78%
Macro F1 Score: 0.8760
FNN Epoch 1: loss = 0.5547
FNN Epoch 2: loss = 0.4028
FNN Epoch 3: loss = 0.3557
FNN Epoch 4: loss = 0.3287
FNN Epoch 5: loss = 0.3106
Test Accuracy: 87.35%
Macro F1 Score: 0.8728
FNN Epoch 1: loss = 0.5345
FNN Epoch 2: loss = 0.3877
FNN Epoch 3: loss = 0.3428
FNN Epoch 4: loss = 0.3162
FNN Epoch 5: loss = 0.2983
Test Accuracy: 87.87%
Macro F1 Score: 0.8781
FNN Epoch 1: loss = 0.6026
FNN Epoch 2: loss = 0.4247
FNN Epoch 3: loss = 0.3849
FNN Epoch 4: loss = 0.3544
FNN Epoch 5: loss = 0.3342
Test Accuracy: 86.89%
Macro F1 Score: 0.8676
FNN Epoch 1: loss = 0.5496
FNN Epoch 2: loss = 0.3985
FNN Epoch 3: loss = 0.3574
FNN Epoch 4: loss = 0.3277
FNN Epoch 5: loss = 0.3045
Test Accuracy: 87.16%
Macro F1 Score: 0.8700
FNN Epoch 1: loss = 0.5537
FNN Epoch 2: loss = 0.3964
FNN Epoch 3: loss = 0.3555
FNN Epoch 4: loss = 0.3250
FNN Epoch 5: loss = 0.3043
Test Accuracy: 86.52%
Macro F1 Score: 0.8607
FNN Epoch 1: loss = 0.5199
FNN Epoch 2: loss = 0.3781
FNN Epoch 3: loss = 0.3376
FNN Epoch 4: loss = 0.3090
FNN Epoch 5: loss = 0.2903
Test Accuracy: 86.14%
Macro F1 Score: 0.8651
FNN Epoch 1: loss = 0.5423
FNN Epoch 2: loss = 0.3924
FNN Epoch 3: loss = 0.3503
FNN Epoch 4: loss = 0.3227
FNN Epoch 5: loss = 0.3023
Test Accuracy: 87.04%
Macro F1 Score: 0.8698
FNN Epoch 1: loss = 0.5308
FNN Epoch 2: loss = 0.3882
FNN Epoch 3: loss = 0.3439
FNN Epoch 4: loss = 0.3196
FNN Epoch 5: loss = 0.2953
Test Accuracy: 87.78%
Macro F1 Score: 0.8770
FNN Epoch 1: loss = 0.6554
FNN Epoch 2: loss = 0.4416
FNN Epoch 3: loss = 0.3996
FNN Epoch 4: loss = 0.3730
FNN Epoch 5: loss = 0.3523
Test Accuracy: 86.74%
Macro F1 Score: 0.8670
FNN Epoch 1: loss = 0.5975
FNN Epoch 2: loss = 0.4166
FNN Epoch 3: loss = 0.3752
FNN Epoch 4: loss = 0.3500
FNN Epoch 5: loss = 0.3280
Test Accuracy: 86.97%
Macro F1 Score: 0.8686
FNN Epoch 1: loss = 0.5616
FNN Epoch 2: loss = 0.3995
FNN Epoch 3: loss = 0.3581
FNN Epoch 4: loss = 0.3303
FNN Epoch 5: loss = 0.3112
Test Accuracy: 86.01%
Macro F1 Score: 0.8624
FNN Epoch 1: loss = 0.6865
FNN Epoch 2: loss = 0.4488
FNN Epoch 3: loss = 0.4080
FNN Epoch 4: loss = 0.3788
FNN Epoch 5: loss = 0.3601
Test Accuracy: 86.15%
Macro F1 Score: 0.8595
FNN Epoch 1: loss = 0.5225
FNN Epoch 2: loss = 0.3804
FNN Epoch 3: loss = 0.3394
FNN Epoch 4: loss = 0.3153
FNN Epoch 5: loss = 0.2945
Test Accuracy: 87.46%
Macro F1 Score: 0.8747
FNN Epoch 1: loss = 0.5712
FNN Epoch 2: loss = 0.4067
FNN Epoch 3: loss = 0.3648
FNN Epoch 4: loss = 0.3370
FNN Epoch 5: loss = 0.3139
Test Accuracy: 86.31%
Macro F1 Score: 0.8640
FNN Epoch 1: loss = 0.5433
FNN Epoch 2: loss = 0.3874
FNN Epoch 3: loss = 0.3427
FNN Epoch 4: loss = 0.3149
FNN Epoch 5: loss = 0.2957
Test Accuracy: 85.75%
Macro F1 Score: 0.8571
FNN Epoch 1: loss = 0.4929
FNN Epoch 2: loss = 0.3677
FNN Epoch 3: loss = 0.3292
FNN Epoch 4: loss = 0.3018
FNN Epoch 5: loss = 0.2864
Test Accuracy: 87.70%
Macro F1 Score: 0.8764
FNN Epoch 1: loss = 0.4886
FNN Epoch 2: loss = 0.3673
FNN Epoch 3: loss = 0.3274
FNN Epoch 4: loss = 0.3068
FNN Epoch 5: loss = 0.2878
Test Accuracy: 87.52%
Macro F1 Score: 0.8737
FNN Epoch 1: loss = 0.4845
FNN Epoch 2: loss = 0.3652
FNN Epoch 3: loss = 0.3275
FNN Epoch 4: loss = 0.3064
FNN Epoch 5: loss = 0.2892
Test Accuracy: 87.33%
Macro F1 Score: 0.8724
FNN Epoch 1: loss = 0.5210
FNN Epoch 2: loss = 0.3791
FNN Epoch 3: loss = 0.3399
FNN Epoch 4: loss = 0.3124
FNN Epoch 5: loss = 0.2948
Test Accuracy: 87.62%
Macro F1 Score: 0.8742
FNN Epoch 1: loss = 0.5101
FNN Epoch 2: loss = 0.3730
FNN Epoch 3: loss = 0.3291
FNN Epoch 4: loss = 0.3068
FNN Epoch 5: loss = 0.2881
Test Accuracy: 87.65%
Macro F1 Score: 0.8758
FNN Epoch 1: loss = 0.6214
FNN Epoch 2: loss = 0.4269
FNN Epoch 3: loss = 0.3902
FNN Epoch 4: loss = 0.3595
FNN Epoch 5: loss = 0.3372
Test Accuracy: 86.25%
Macro F1 Score: 0.8624
FNN Epoch 1: loss = 0.5439
FNN Epoch 2: loss = 0.3959
FNN Epoch 3: loss = 0.3516
FNN Epoch 4: loss = 0.3219
FNN Epoch 5: loss = 0.3036
Test Accuracy: 87.66%
Macro F1 Score: 0.8760
FNN Epoch 1: loss = 0.5466
FNN Epoch 2: loss = 0.3964
FNN Epoch 3: loss = 0.3551
FNN Epoch 4: loss = 0.3275
FNN Epoch 5: loss = 0.3086
Test Accuracy: 86.08%
Macro F1 Score: 0.8592
FNN Epoch 1: loss = 0.4928
FNN Epoch 2: loss = 0.3662
FNN Epoch 3: loss = 0.3298
FNN Epoch 4: loss = 0.3058
FNN Epoch 5: loss = 0.2842
Test Accuracy: 86.82%
Macro F1 Score: 0.8693
Best Parameters for FNN: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 304, 'learning_rate': 0.0010717942740929352}
Best Accuracy: 87.95

[I 2025-03-28 14:08:49,504] A new study created in memory with name: no-name-983a1dcb-4266-42aa-ba3a-6807a1759ec8
[I 2025-03-28 14:09:04,255] Trial 0 finished with value: 87.28 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 316, 'learning_rate': 0.0019643108659146637}. Best is trial 0 with value: 87.28.
[I 2025-03-28 14:09:16,162] Trial 1 finished with value: 85.83 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'fnn_hidden': 218, 'learning_rate': 0.0005743084788275644}. Best is trial 0 with value: 87.28.
[I 2025-03-28 14:09:27,990] Trial 2 finished with value: 87.59 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'fnn_hidden': 371, 'learning_rate': 0.0019863794468380712}. Best is trial 2 with value: 87.59.
[I 2025-03-28 14:09:40,429] Trial 3 finished with value: 86.81 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'fnn_hidden': 286, 'learning_rate': 0.000712170280473209}. Best is trial 2 with value: 87.59.
[I 2025-03-28 14:09:53,781] Trial 4 finished with value: 87.62 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'fnn_hidden': 334, 'learning_rate': 0.0020136851145242008}. Best is trial 4 with value: 87.62.
[I 2025-03-28 14:10:07,119] Trial 5 finished with value: 86.71 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'fnn_hidden': 234, 'learning_rate': 0.000879950233252207}. Best is trial 4 with value: 87.62.
[I 2025-03-28 14:25:49,791] Trial 6 finished with value: 87.95 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 304, 'learning_rate': 0.0010717942740929352}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:26:02,383] Trial 7 finished with value: 83.61 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'fnn_hidden': 343, 'learning_rate': 0.00010498879450062599}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:26:13,821] Trial 8 finished with value: 86.77 and parameters: {'batch_size': 256, 'num_classifier_epochs': 5, 'fnn_hidden': 234, 'learning_rate': 0.002295260294245649}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:26:25,825] Trial 9 finished with value: 85.08 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'fnn_hidden': 325, 'learning_rate': 0.00024344110913700564}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:26:45,755] Trial 10 finished with value: 86.99 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 278, 'learning_rate': 0.0012933878073006246}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:27:05,756] Trial 11 finished with value: 87.77 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 384, 'learning_rate': 0.001456679915410512}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:27:25,685] Trial 12 finished with value: 87.48 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 377, 'learning_rate': 0.00134565096431046}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:27:42,214] Trial 13 finished with value: 85.96 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 267, 'learning_rate': 0.0013802805587902047}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:27:58,589] Trial 14 finished with value: 87.03 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 197, 'learning_rate': 0.0010918734361520083}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:28:15,043] Trial 15 finished with value: 87.53 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 354, 'learning_rate': 0.0016476428641274438}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:28:34,936] Trial 16 finished with value: 87.5 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 295, 'learning_rate': 0.0016223464008806217}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:28:49,545] Trial 17 finished with value: 87.29 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 384, 'learning_rate': 0.0010056153978433512}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:29:03,976] Trial 18 finished with value: 86.39 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 262, 'learning_rate': 0.0005037017776003777}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:29:24,158] Trial 19 finished with value: 87.66 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 305, 'learning_rate': 0.0016580598066331673}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:29:40,807] Trial 20 finished with value: 86.23 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 355, 'learning_rate': 0.002390707562410727}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:30:00,723] Trial 21 finished with value: 87.66 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 306, 'learning_rate': 0.0015819689156265399}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:30:20,585] Trial 22 finished with value: 87.65 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 309, 'learning_rate': 0.0017706227694859965}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:30:37,245] Trial 23 finished with value: 87.11 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 257, 'learning_rate': 0.0011721567248532275}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:30:57,144] Trial 24 finished with value: 86.11 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 297, 'learning_rate': 0.0014901384184530485}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:31:11,770] Trial 25 finished with value: 87.78 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 347, 'learning_rate': 0.0008857376095970037}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:31:26,472] Trial 26 finished with value: 87.35 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 362, 'learning_rate': 0.0008330810395781131}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:31:43,017] Trial 27 finished with value: 87.87 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 342, 'learning_rate': 0.0009495722354665602}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:31:57,652] Trial 28 finished with value: 86.89 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 340, 'learning_rate': 0.0005170836542807583}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:32:12,291] Trial 29 finished with value: 87.16 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 321, 'learning_rate': 0.0009883480804708245}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:32:28,922] Trial 30 finished with value: 86.52 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 330, 'learning_rate': 0.0007201210827906603}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:32:45,538] Trial 31 finished with value: 86.14 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 364, 'learning_rate': 0.0011965834722640672}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:33:02,031] Trial 32 finished with value: 87.04 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 348, 'learning_rate': 0.0008632040329805744}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:33:21,870] Trial 33 finished with value: 87.78 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 371, 'learning_rate': 0.0006580877176856504}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:33:36,422] Trial 34 finished with value: 86.74 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 365, 'learning_rate': 0.00034264315176449806}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:33:49,798] Trial 35 finished with value: 86.97 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'fnn_hidden': 339, 'learning_rate': 0.0006735523042490962}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:49:51,445] Trial 36 finished with value: 86.01 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 371, 'learning_rate': 0.0006317392699132341}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:50:54,737] Trial 37 finished with value: 86.15 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'fnn_hidden': 316, 'learning_rate': 0.00041861541457776253}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:52:31,018] Trial 38 finished with value: 87.46 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 349, 'learning_rate': 0.0008043968547649064}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:57:10,840] Trial 39 finished with value: 86.31 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'fnn_hidden': 333, 'learning_rate': 0.0009800005538671466}. Best is trial 6 with value: 87.95.
[I 2025-03-28 14:58:36,766] Trial 40 finished with value: 85.75 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'fnn_hidden': 324, 'learning_rate': 0.0011877036439728564}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:06:40,962] Trial 41 finished with value: 87.7 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 383, 'learning_rate': 0.0014521830179726831}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:07:01,028] Trial 42 finished with value: 87.52 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 373, 'learning_rate': 0.001844578972387457}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:07:20,882] Trial 43 finished with value: 87.33 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 355, 'learning_rate': 0.0021212829675613725}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:07:38,719] Trial 44 finished with value: 87.62 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 374, 'learning_rate': 0.001091989921026165}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:08:02,511] Trial 45 finished with value: 87.65 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 361, 'learning_rate': 0.0012472755708203321}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:08:15,197] Trial 46 finished with value: 86.25 and parameters: {'batch_size': 256, 'num_classifier_epochs': 5, 'fnn_hidden': 279, 'learning_rate': 0.0009346075763025649}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:08:31,922] Trial 47 finished with value: 87.66 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 345, 'learning_rate': 0.0007707965198842552}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:08:48,744] Trial 48 finished with value: 86.08 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'fnn_hidden': 243, 'learning_rate': 0.0010686543058994109}. Best is trial 6 with value: 87.95.
[I 2025-03-28 15:09:08,744] Trial 49 finished with value: 86.82 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'fnn_hidden': 382, 'learning_rate': 0.0013456789092006166}. Best is trial 6 with value: 87.95.

Test Accuracy by FNN Hidden Units

Model 3: Convolutional Neural Network on Fashion MNIST Data
Base code for CNN structure borrowed from Kaggle

Click to Show Code and Output

from sklearn.metrics import f1_score

CLASSIFIER = "CNN"  # Change for FNN, LogisticRegression, or CNN

# Define CNN model
class FashionCNN(nn.Module):
    def __init__(self, filters1, filters2, kernel1, kernel2):
        super(FashionCNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=filters1, kernel_size=kernel1, padding=1),
            nn.BatchNorm2d(filters1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=filters1, out_channels=filters2, kernel_size=kernel2),
            nn.BatchNorm2d(filters2),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        self.fc1 = None
        self.drop = nn.Dropout2d(0.25)
        self.fc2 = nn.Linear(in_features=600, out_features=120)
        self.fc3 = nn.Linear(in_features=120, out_features=10)
        

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        if self.fc1 is None:
            self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)
        out = self.fc1(out)
        out = self.drop(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out

        # Dynamically calculate flattened size
        out = out.view(out.size(0), -1)  # Flatten
        if self.fc1 is None:
            self.fc1 = nn.Linear(out.shape[1], 600).to(x.device)  # ✅ Update FC layer dynamically

        out = self.fc1(out)
        out = self.drop(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out




# Define Optuna objective function
def objective(trial):
        # Set MLflow experiment name
    if CLASSIFIER == "LogisticRegression":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-noRBM")
    elif CLASSIFIER == "FNN":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-noRBM")
    elif CLASSIFIER == "CNN":
        experiment = mlflow.set_experiment("new-pytorch-fmnist-cnn-noRBM")
    batch_size = trial.suggest_int("batch_size", 64, 256, step=32)
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

    mlflow.start_run(experiment_id=experiment.experiment_id)
    num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5) 
    mlflow.log_param("num_classifier_epochs", num_classifier_epochs)

    if CLASSIFIER == "FNN":
        hidden_size = trial.suggest_int("fnn_hidden", 192, 384)
        learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)

        mlflow.log_param("classifier", "FNN")
        mlflow.log_param("fnn_hidden", hidden_size)
        mlflow.log_param("learning_rate", learning_rate)

        model = nn.Sequential(
            nn.Linear(784, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, 10)
        ).to(device)

        optimizer = optim.Adam(model.parameters(), lr=learning_rate)

    elif CLASSIFIER == "CNN":
        filters1 = trial.suggest_int("filters1", 16, 64, step=16)
        filters2 = trial.suggest_int("filters2", 32, 128, step=32)
        kernel1 = trial.suggest_int("kernel1", 3, 5)
        kernel2 = trial.suggest_int("kernel2", 3, 5)
        learning_rate = trial.suggest_float("learning_rate", 0.0001, 0.0025)

        mlflow.log_param("classifier", "CNN")
        mlflow.log_param("filters1", filters1)
        mlflow.log_param("filters2", filters2)
        mlflow.log_param("kernel1", kernel1)
        mlflow.log_param("kernel2", kernel2)
        mlflow.log_param("learning_rate", learning_rate)

        model = FashionCNN(filters1, filters2, kernel1, kernel2).to(device)
        optimizer = optim.Adam(model.parameters(), lr=learning_rate)

      
    elif CLASSIFIER == "LogisticRegression":
        mlflow.log_param("classifier", "LogisticRegression")
    
        # Prepare data for Logistic Regression (Flatten 28x28 images to 784 features)
        train_features = train_dataset.data.view(-1, 784).numpy()
        train_labels = train_dataset.targets.numpy()
        test_features = test_dataset.data.view(-1, 784).numpy()
        test_labels = test_dataset.targets.numpy()
    
        # Normalize the pixel values to [0,1] for better convergence
        train_features = train_features / 255.0
        test_features = test_features / 255.0
    
    
        C = trial.suggest_float("C", 0.01, 10.0, log=True)  
        solver = "saga" 
    
        model = LogisticRegression(C=C, max_iter=num_classifier_epochs, solver=solver)
        model.fit(train_features, train_labels)
    
    
        predictions = model.predict(test_features)
        accuracy = accuracy_score(test_labels, predictions) * 100
        
        macro_f1 = f1_score(test_labels, predictions, average="macro") #for f1
        print(f"Logistic Regression Test Accuracy: {accuracy:.2f}%")
        print(f"Macro F1 Score: {macro_f1:.4f}") #for f1
    
        mlflow.log_param("C", C)
        mlflow.log_metric("test_accuracy", accuracy)
        mlflow.log_metric("macro_f1", macro_f1) #for f1
        mlflow.end_run()
        return accuracy

    # Training Loop for FNN and CNN
    criterion = nn.CrossEntropyLoss()

    model.train()
    for epoch in range(num_classifier_epochs):
        running_loss = 0.0
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))

            optimizer.zero_grad()
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        print(f"{CLASSIFIER} Epoch {epoch+1}: loss = {running_loss / len(train_loader):.4f}")

    # Model Evaluation
    model.eval()
    correct, total = 0, 0
    all_preds = []   # for f1
    all_labels = [] 
    with torch.no_grad():
        for images, labels in test_loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images) if CLASSIFIER == "CNN" else model(images.view(images.size(0), -1))
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            all_preds.extend(predicted.cpu().numpy())   #for f1
            all_labels.extend(labels.cpu().numpy()) #for f1

    accuracy = 100 * correct / total
    macro_f1 = f1_score(all_labels, all_preds, average="macro") #for f1
    print(f"Test Accuracy: {accuracy:.2f}%")
    print(f"Macro F1 Score: {macro_f1:.4f}") #for f1

    mlflow.log_metric("test_accuracy", accuracy)
    mlflow.log_metric("macro_f1", macro_f1) #for f1
    mlflow.end_run()
    return accuracy

if __name__ == "__main__":
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=50)
    print(f"Best Parameters for {CLASSIFIER}:", study.best_params)
    print("Best Accuracy:", study.best_value)

CNN Epoch 1: loss = 0.4142
CNN Epoch 2: loss = 0.3038
CNN Epoch 3: loss = 0.2718
CNN Epoch 4: loss = 0.2433
CNN Epoch 5: loss = 0.2250
Test Accuracy: 89.01%
Macro F1 Score: 0.8920
CNN Epoch 1: loss = 0.4187
CNN Epoch 2: loss = 0.2990
CNN Epoch 3: loss = 0.2635
CNN Epoch 4: loss = 0.2406
CNN Epoch 5: loss = 0.2229
Test Accuracy: 89.64%
Macro F1 Score: 0.8964
CNN Epoch 1: loss = 0.7600
CNN Epoch 2: loss = 0.4102
CNN Epoch 3: loss = 0.3472
CNN Epoch 4: loss = 0.3177
CNN Epoch 5: loss = 0.2972
Test Accuracy: 88.70%
Macro F1 Score: 0.8863
CNN Epoch 1: loss = 0.4309
CNN Epoch 2: loss = 0.3029
CNN Epoch 3: loss = 0.2658
CNN Epoch 4: loss = 0.2438
CNN Epoch 5: loss = 0.2262
Test Accuracy: 90.06%
Macro F1 Score: 0.9009
CNN Epoch 1: loss = 0.4653
CNN Epoch 2: loss = 0.3246
CNN Epoch 3: loss = 0.2935
CNN Epoch 4: loss = 0.2697
CNN Epoch 5: loss = 0.2527
Test Accuracy: 89.10%
Macro F1 Score: 0.8901
CNN Epoch 1: loss = 0.6973
CNN Epoch 2: loss = 0.3831
CNN Epoch 3: loss = 0.3287
CNN Epoch 4: loss = 0.2999
CNN Epoch 5: loss = 0.2794
Test Accuracy: 89.05%
Macro F1 Score: 0.8893
CNN Epoch 1: loss = 0.5288
CNN Epoch 2: loss = 0.3632
CNN Epoch 3: loss = 0.3297
CNN Epoch 4: loss = 0.3076
CNN Epoch 5: loss = 0.2952
Test Accuracy: 87.87%
Macro F1 Score: 0.8755
CNN Epoch 1: loss = 0.4134
CNN Epoch 2: loss = 0.2940
CNN Epoch 3: loss = 0.2597
CNN Epoch 4: loss = 0.2390
CNN Epoch 5: loss = 0.2225
Test Accuracy: 89.62%
Macro F1 Score: 0.8967
CNN Epoch 1: loss = 0.5063
CNN Epoch 2: loss = 0.3385
CNN Epoch 3: loss = 0.2986
CNN Epoch 4: loss = 0.2803
CNN Epoch 5: loss = 0.2600
Test Accuracy: 88.94%
Macro F1 Score: 0.8894
CNN Epoch 1: loss = 0.4016
CNN Epoch 2: loss = 0.2801
CNN Epoch 3: loss = 0.2490
CNN Epoch 4: loss = 0.2251
CNN Epoch 5: loss = 0.2067
Test Accuracy: 90.66%
Macro F1 Score: 0.9066
CNN Epoch 1: loss = 0.4207
CNN Epoch 2: loss = 0.3202
CNN Epoch 3: loss = 0.2878
CNN Epoch 4: loss = 0.2679
CNN Epoch 5: loss = 0.2524
Test Accuracy: 89.03%
Macro F1 Score: 0.8876
CNN Epoch 1: loss = 0.4363
CNN Epoch 2: loss = 0.3068
CNN Epoch 3: loss = 0.2734
CNN Epoch 4: loss = 0.2506
CNN Epoch 5: loss = 0.2341
Test Accuracy: 90.32%
Macro F1 Score: 0.9029
CNN Epoch 1: loss = 0.4417
CNN Epoch 2: loss = 0.3104
CNN Epoch 3: loss = 0.2801
CNN Epoch 4: loss = 0.2627
CNN Epoch 5: loss = 0.2508
Test Accuracy: 90.57%
Macro F1 Score: 0.9036
CNN Epoch 1: loss = 0.4490
CNN Epoch 2: loss = 0.3182
CNN Epoch 3: loss = 0.2877
CNN Epoch 4: loss = 0.2672
CNN Epoch 5: loss = 0.2551
Test Accuracy: 90.72%
Macro F1 Score: 0.9071
CNN Epoch 1: loss = 0.4475
CNN Epoch 2: loss = 0.3125
CNN Epoch 3: loss = 0.2839
CNN Epoch 4: loss = 0.2675
CNN Epoch 5: loss = 0.2553
Test Accuracy: 89.95%
Macro F1 Score: 0.8995
CNN Epoch 1: loss = 0.4153
CNN Epoch 2: loss = 0.3026
CNN Epoch 3: loss = 0.2689
CNN Epoch 4: loss = 0.2477
CNN Epoch 5: loss = 0.2294
Test Accuracy: 90.06%
Macro F1 Score: 0.8993
CNN Epoch 1: loss = 0.4508
CNN Epoch 2: loss = 0.3101
CNN Epoch 3: loss = 0.2834
CNN Epoch 4: loss = 0.2615
CNN Epoch 5: loss = 0.2463
Test Accuracy: 90.46%
Macro F1 Score: 0.9057
CNN Epoch 1: loss = 0.5764
CNN Epoch 2: loss = 0.3532
CNN Epoch 3: loss = 0.3157
CNN Epoch 4: loss = 0.2901
CNN Epoch 5: loss = 0.2754
Test Accuracy: 88.71%
Macro F1 Score: 0.8865
CNN Epoch 1: loss = 0.4089
CNN Epoch 2: loss = 0.3025
CNN Epoch 3: loss = 0.2707
CNN Epoch 4: loss = 0.2446
CNN Epoch 5: loss = 0.2278
Test Accuracy: 90.61%
Macro F1 Score: 0.9052
CNN Epoch 1: loss = 0.4471
CNN Epoch 2: loss = 0.3439
CNN Epoch 3: loss = 0.3163
CNN Epoch 4: loss = 0.2981
CNN Epoch 5: loss = 0.2857
Test Accuracy: 88.90%
Macro F1 Score: 0.8888
CNN Epoch 1: loss = 0.4267
CNN Epoch 2: loss = 0.3039
CNN Epoch 3: loss = 0.2697
CNN Epoch 4: loss = 0.2476
CNN Epoch 5: loss = 0.2324
Test Accuracy: 90.20%
Macro F1 Score: 0.9004
CNN Epoch 1: loss = 0.4007
CNN Epoch 2: loss = 0.2934
CNN Epoch 3: loss = 0.2662
CNN Epoch 4: loss = 0.2444
CNN Epoch 5: loss = 0.2286
Test Accuracy: 90.59%
Macro F1 Score: 0.9053
CNN Epoch 1: loss = 0.4259
CNN Epoch 2: loss = 0.3050
CNN Epoch 3: loss = 0.2688
CNN Epoch 4: loss = 0.2507
CNN Epoch 5: loss = 0.2319
Test Accuracy: 89.37%
Macro F1 Score: 0.8954
CNN Epoch 1: loss = 0.4184
CNN Epoch 2: loss = 0.3066
CNN Epoch 3: loss = 0.2757
CNN Epoch 4: loss = 0.2559
CNN Epoch 5: loss = 0.2391
Test Accuracy: 89.50%
Macro F1 Score: 0.8956
CNN Epoch 1: loss = 0.4162
CNN Epoch 2: loss = 0.2946
CNN Epoch 3: loss = 0.2613
CNN Epoch 4: loss = 0.2372
CNN Epoch 5: loss = 0.2225
Test Accuracy: 90.40%
Macro F1 Score: 0.9034
CNN Epoch 1: loss = 0.4351
CNN Epoch 2: loss = 0.3094
CNN Epoch 3: loss = 0.2759
CNN Epoch 4: loss = 0.2547
CNN Epoch 5: loss = 0.2396
Test Accuracy: 90.47%
Macro F1 Score: 0.9028
CNN Epoch 1: loss = 0.4139
CNN Epoch 2: loss = 0.3120
CNN Epoch 3: loss = 0.2817
CNN Epoch 4: loss = 0.2602
CNN Epoch 5: loss = 0.2470
Test Accuracy: 90.54%
Macro F1 Score: 0.9041
CNN Epoch 1: loss = 0.5075
CNN Epoch 2: loss = 0.3379
CNN Epoch 3: loss = 0.3067
CNN Epoch 4: loss = 0.2872
CNN Epoch 5: loss = 0.2717
Test Accuracy: 89.07%
Macro F1 Score: 0.8904
CNN Epoch 1: loss = 0.4301
CNN Epoch 2: loss = 0.3226
CNN Epoch 3: loss = 0.2851
CNN Epoch 4: loss = 0.2600
CNN Epoch 5: loss = 0.2436
Test Accuracy: 88.91%
Macro F1 Score: 0.8865
CNN Epoch 1: loss = 0.4362
CNN Epoch 2: loss = 0.3067
CNN Epoch 3: loss = 0.2723
CNN Epoch 4: loss = 0.2491
CNN Epoch 5: loss = 0.2273
Test Accuracy: 88.65%
Macro F1 Score: 0.8873
CNN Epoch 1: loss = 0.4248
CNN Epoch 2: loss = 0.3153
CNN Epoch 3: loss = 0.2845
CNN Epoch 4: loss = 0.2639
CNN Epoch 5: loss = 0.2519
Test Accuracy: 90.41%
Macro F1 Score: 0.9027
CNN Epoch 1: loss = 0.4092
CNN Epoch 2: loss = 0.2970
CNN Epoch 3: loss = 0.2692
CNN Epoch 4: loss = 0.2482
CNN Epoch 5: loss = 0.2305
Test Accuracy: 90.80%
Macro F1 Score: 0.9069
CNN Epoch 1: loss = 0.4054
CNN Epoch 2: loss = 0.2987
CNN Epoch 3: loss = 0.2691
CNN Epoch 4: loss = 0.2478
CNN Epoch 5: loss = 0.2322
Test Accuracy: 90.07%
Macro F1 Score: 0.9006
CNN Epoch 1: loss = 0.4314
CNN Epoch 2: loss = 0.3167
CNN Epoch 3: loss = 0.2855
CNN Epoch 4: loss = 0.2675
CNN Epoch 5: loss = 0.2541
Test Accuracy: 89.69%
Macro F1 Score: 0.8972
CNN Epoch 1: loss = 0.4089
CNN Epoch 2: loss = 0.2978
CNN Epoch 3: loss = 0.2682
CNN Epoch 4: loss = 0.2434
CNN Epoch 5: loss = 0.2267
Test Accuracy: 89.76%
Macro F1 Score: 0.8966
CNN Epoch 1: loss = 0.4152
CNN Epoch 2: loss = 0.3107
CNN Epoch 3: loss = 0.2766
CNN Epoch 4: loss = 0.2571
CNN Epoch 5: loss = 0.2399
Test Accuracy: 90.07%
Macro F1 Score: 0.8994
CNN Epoch 1: loss = 0.4580
CNN Epoch 2: loss = 0.3330
CNN Epoch 3: loss = 0.2997
CNN Epoch 4: loss = 0.2813
CNN Epoch 5: loss = 0.2663
Test Accuracy: 89.50%
Macro F1 Score: 0.8910
CNN Epoch 1: loss = 0.4317
CNN Epoch 2: loss = 0.3087
CNN Epoch 3: loss = 0.2760
CNN Epoch 4: loss = 0.2483
CNN Epoch 5: loss = 0.2314
Test Accuracy: 89.52%
Macro F1 Score: 0.8963
CNN Epoch 1: loss = 0.4271
CNN Epoch 2: loss = 0.3116
CNN Epoch 3: loss = 0.2782
CNN Epoch 4: loss = 0.2563
CNN Epoch 5: loss = 0.2451
Test Accuracy: 90.13%
Macro F1 Score: 0.8997
CNN Epoch 1: loss = 0.4143
CNN Epoch 2: loss = 0.3035
CNN Epoch 3: loss = 0.2696
CNN Epoch 4: loss = 0.2497
CNN Epoch 5: loss = 0.2333
Test Accuracy: 90.29%
Macro F1 Score: 0.9026
CNN Epoch 1: loss = 0.4845
CNN Epoch 2: loss = 0.3127
CNN Epoch 3: loss = 0.2779
CNN Epoch 4: loss = 0.2596
CNN Epoch 5: loss = 0.2401
Test Accuracy: 89.62%
Macro F1 Score: 0.8953
CNN Epoch 1: loss = 0.4058
CNN Epoch 2: loss = 0.3002
CNN Epoch 3: loss = 0.2676
CNN Epoch 4: loss = 0.2463
CNN Epoch 5: loss = 0.2321
Test Accuracy: 90.62%
Macro F1 Score: 0.9057
CNN Epoch 1: loss = 0.4072
CNN Epoch 2: loss = 0.2997
CNN Epoch 3: loss = 0.2663
CNN Epoch 4: loss = 0.2460
CNN Epoch 5: loss = 0.2309
Test Accuracy: 89.75%
Macro F1 Score: 0.8966
CNN Epoch 1: loss = 0.4278
CNN Epoch 2: loss = 0.3188
CNN Epoch 3: loss = 0.2846
CNN Epoch 4: loss = 0.2606
CNN Epoch 5: loss = 0.2442
Test Accuracy: 89.74%
Macro F1 Score: 0.8971
CNN Epoch 1: loss = 0.4355
CNN Epoch 2: loss = 0.3071
CNN Epoch 3: loss = 0.2800
CNN Epoch 4: loss = 0.2587
CNN Epoch 5: loss = 0.2459
Test Accuracy: 89.94%
Macro F1 Score: 0.9001
CNN Epoch 1: loss = 0.4154
CNN Epoch 2: loss = 0.3004
CNN Epoch 3: loss = 0.2682
CNN Epoch 4: loss = 0.2464
CNN Epoch 5: loss = 0.2292
Test Accuracy: 89.21%
Macro F1 Score: 0.8904
CNN Epoch 1: loss = 0.4167
CNN Epoch 2: loss = 0.3086
CNN Epoch 3: loss = 0.2803
CNN Epoch 4: loss = 0.2568
CNN Epoch 5: loss = 0.2417
Test Accuracy: 90.07%
Macro F1 Score: 0.9000
CNN Epoch 1: loss = 0.4117
CNN Epoch 2: loss = 0.3026
CNN Epoch 3: loss = 0.2720
CNN Epoch 4: loss = 0.2514
CNN Epoch 5: loss = 0.2347
Test Accuracy: 90.46%
Macro F1 Score: 0.9016
CNN Epoch 1: loss = 0.4738
CNN Epoch 2: loss = 0.3234
CNN Epoch 3: loss = 0.2905
CNN Epoch 4: loss = 0.2703
CNN Epoch 5: loss = 0.2589
Test Accuracy: 90.09%
Macro F1 Score: 0.8998
CNN Epoch 1: loss = 0.3981
CNN Epoch 2: loss = 0.2907
CNN Epoch 3: loss = 0.2582
CNN Epoch 4: loss = 0.2387
CNN Epoch 5: loss = 0.2174
Test Accuracy: 90.19%
Macro F1 Score: 0.9036
Best Parameters for CNN: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0015478685740995355}
Best Accuracy: 90.8

[I 2025-03-28 15:09:09,142] A new study created in memory with name: no-name-50d0bad0-9df4-4c8f-ac1e-663e44131eb2
[I 2025-03-28 15:10:00,337] Trial 0 finished with value: 89.01 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 128, 'kernel1': 4, 'kernel2': 5, 'learning_rate': 0.0015248505233275051}. Best is trial 0 with value: 89.01.
[I 2025-03-28 15:10:47,448] Trial 1 finished with value: 89.64 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 128, 'kernel1': 4, 'kernel2': 4, 'learning_rate': 0.0012229501836420783}. Best is trial 1 with value: 89.64.
[I 2025-03-28 15:11:21,298] Trial 2 finished with value: 88.7 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'filters1': 64, 'filters2': 32, 'kernel1': 5, 'kernel2': 4, 'learning_rate': 0.00013313360782682066}. Best is trial 1 with value: 89.64.
[I 2025-03-28 15:12:04,707] Trial 3 finished with value: 90.06 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'filters1': 64, 'filters2': 96, 'kernel1': 5, 'kernel2': 4, 'learning_rate': 0.0013252696356154724}. Best is trial 3 with value: 90.06.
[I 2025-03-28 15:12:37,933] Trial 4 finished with value: 89.1 and parameters: {'batch_size': 256, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 96, 'kernel1': 5, 'kernel2': 3, 'learning_rate': 0.0016266946027195346}. Best is trial 3 with value: 90.06.
[I 2025-03-28 15:13:07,637] Trial 5 finished with value: 89.05 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 96, 'kernel1': 4, 'kernel2': 4, 'learning_rate': 0.00017858037364493448}. Best is trial 3 with value: 90.06.
[I 2025-03-28 15:13:33,636] Trial 6 finished with value: 87.87 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 32, 'kernel1': 4, 'kernel2': 5, 'learning_rate': 0.0007601148742759145}. Best is trial 3 with value: 90.06.
[I 2025-03-28 15:14:12,029] Trial 7 finished with value: 89.62 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0020459440534046185}. Best is trial 3 with value: 90.06.
[I 2025-03-28 15:14:42,736] Trial 8 finished with value: 88.94 and parameters: {'batch_size': 192, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 32, 'kernel1': 4, 'kernel2': 5, 'learning_rate': 0.0006940093332337714}. Best is trial 3 with value: 90.06.
[I 2025-03-28 15:15:35,761] Trial 9 finished with value: 90.66 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 128, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0014227934892389901}. Best is trial 9 with value: 90.66.
[I 2025-03-28 15:16:23,120] Trial 10 finished with value: 89.03 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 128, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.0023379932249639446}. Best is trial 9 with value: 90.66.
[I 2025-03-28 15:17:07,979] Trial 11 finished with value: 90.32 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 64, 'filters2': 96, 'kernel1': 3, 'kernel2': 4, 'learning_rate': 0.0011412008327813165}. Best is trial 9 with value: 90.66.
[I 2025-03-28 15:17:43,163] Trial 12 finished with value: 90.57 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 64, 'filters2': 64, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.0008932519608388764}. Best is trial 9 with value: 90.66.
[I 2025-03-28 15:18:20,994] Trial 13 finished with value: 90.72 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.000759338062230861}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:19:08,668] Trial 14 finished with value: 89.95 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.0005171672224923642}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:19:46,931] Trial 15 finished with value: 90.06 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0019349248934961896}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:20:19,823] Trial 16 finished with value: 90.46 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 128, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.000990739247525854}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:20:43,224] Trial 17 finished with value: 88.71 and parameters: {'batch_size': 256, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 96, 'kernel1': 3, 'kernel2': 4, 'learning_rate': 0.0005322502897563381}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:21:33,148] Trial 18 finished with value: 90.61 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.001620196730554928}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:22:11,027] Trial 19 finished with value: 88.9 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 32, 'kernel1': 4, 'kernel2': 3, 'learning_rate': 0.001839707721726052}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:22:53,680] Trial 20 finished with value: 90.2 and parameters: {'batch_size': 224, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 128, 'kernel1': 3, 'kernel2': 4, 'learning_rate': 0.0024889796133159615}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:23:43,801] Trial 21 finished with value: 90.59 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0015269816451829285}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:24:14,489] Trial 22 finished with value: 89.37 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0016903502364860711}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:24:59,127] Trial 23 finished with value: 89.5 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0013748872433316651}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:25:38,067] Trial 24 finished with value: 90.4 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 96, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0010741522450846625}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:26:13,188] Trial 25 finished with value: 90.47 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 64, 'kernel1': 4, 'kernel2': 4, 'learning_rate': 0.001336707440991061}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:26:59,105] Trial 26 finished with value: 90.54 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 32, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.002210539142056378}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:27:36,611] Trial 27 finished with value: 89.07 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 96, 'kernel1': 3, 'kernel2': 4, 'learning_rate': 0.00039540026208312746}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:28:16,775] Trial 28 finished with value: 88.91 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 64, 'kernel1': 4, 'kernel2': 5, 'learning_rate': 0.0017796312115365507}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:28:52,596] Trial 29 finished with value: 88.65 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 128, 'kernel1': 4, 'kernel2': 5, 'learning_rate': 0.0014599676720398055}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:29:43,201] Trial 30 finished with value: 90.41 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 64, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.0008604550337401655}. Best is trial 13 with value: 90.72.
[I 2025-03-28 15:30:33,482] Trial 31 finished with value: 90.8 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0015478685740995355}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:31:23,711] Trial 32 finished with value: 90.07 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0015729170015998481}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:32:01,635] Trial 33 finished with value: 89.69 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 32, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.001202790673346334}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:32:42,542] Trial 34 finished with value: 89.76 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.00200718923791879}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:33:32,313] Trial 35 finished with value: 90.07 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 96, 'kernel1': 5, 'kernel2': 4, 'learning_rate': 0.0017332842584397142}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:33:59,965] Trial 36 finished with value: 89.5 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 64, 'kernel1': 3, 'kernel2': 4, 'learning_rate': 0.0012724540542330427}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:34:46,415] Trial 37 finished with value: 89.52 and parameters: {'batch_size': 160, 'num_classifier_epochs': 5, 'filters1': 64, 'filters2': 96, 'kernel1': 4, 'kernel2': 5, 'learning_rate': 0.0014487192931249753}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:35:24,262] Trial 38 finished with value: 90.13 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 32, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0018917869487398652}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:36:06,806] Trial 39 finished with value: 90.29 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 96, 'kernel1': 4, 'kernel2': 4, 'learning_rate': 0.0015880880328107737}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:36:56,119] Trial 40 finished with value: 89.62 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.00025652536965903435}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:37:45,573] Trial 41 finished with value: 90.62 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0014891838000063498}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:38:34,993] Trial 42 finished with value: 89.75 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0016906631687284544}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:39:25,040] Trial 43 finished with value: 89.74 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 5, 'kernel2': 5, 'learning_rate': 0.0013995085634476376}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:39:56,426] Trial 44 finished with value: 89.94 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 32, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0010698644382101088}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:40:34,772] Trial 45 finished with value: 89.21 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0012315042772556648}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:41:19,478] Trial 46 finished with value: 90.07 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 16, 'filters2': 64, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0016297503646640705}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:42:01,265] Trial 47 finished with value: 90.46 and parameters: {'batch_size': 96, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 128, 'kernel1': 3, 'kernel2': 3, 'learning_rate': 0.002079290834484217}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:42:33,040] Trial 48 finished with value: 90.09 and parameters: {'batch_size': 128, 'num_classifier_epochs': 5, 'filters1': 32, 'filters2': 64, 'kernel1': 3, 'kernel2': 4, 'learning_rate': 0.0006739979013646886}. Best is trial 31 with value: 90.8.
[I 2025-03-28 15:43:27,818] Trial 49 finished with value: 90.19 and parameters: {'batch_size': 64, 'num_classifier_epochs': 5, 'filters1': 48, 'filters2': 96, 'kernel1': 3, 'kernel2': 5, 'learning_rate': 0.0014883408074560338}. Best is trial 31 with value: 90.8.

Test Accuracy Based on the Number of Filters in the First Conv2D Layer

Test Accuracy Based on the Number of Filters in the Second Conv2D Layer

Test Accuracy Based on Kernel Size in the First Conv2D Layer

Test Accuracy Based on Kernel Size in the Second Conv2D Layer

Model 4: Logistic Regression on RBM Hidden Features (of Fashion MNIST Data)

Click to Show Code and Output

from sklearn.metrics import accuracy_score, f1_score
CLASSIFIER = 'LogisticRegression'

if CLASSIFIER == 'LogisticRegression':
    experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-withrbm")
else:
    experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-withrbm")


class RBM(nn.Module):
    def __init__(self, n_visible=784, n_hidden=256, k=1):
        super(RBM, self).__init__()
        self.n_visible = n_visible
        self.n_hidden = n_hidden
        # Initialize weights and biases
        self.W = nn.Parameter(torch.randn(n_hidden, n_visible) * 0.1)
        self.v_bias = nn.Parameter(torch.zeros(n_visible))
        self.h_bias = nn.Parameter(torch.zeros(n_hidden))
        self.k = k  # CD-k steps

    def sample_h(self, v):
        # Given visible v, sample hidden h
        p_h = torch.sigmoid(F.linear(v, self.W, self.h_bias))  # p(h=1|v)
        h_sample = torch.bernoulli(p_h)                        # sample Bernoulli
        return p_h, h_sample

    def sample_v(self, h):
        # Given hidden h, sample visible v
        p_v = torch.sigmoid(F.linear(h, self.W.t(), self.v_bias))  # p(v=1|h)
        v_sample = torch.bernoulli(p_v)
        return p_v, v_sample

    def forward(self, v):
        # Perform k steps of contrastive divergence starting from v
        v_k = v.clone()
        for _ in range(self.k):
            _, h_k = self.sample_h(v_k)    # sample hidden from current visible
            _, v_k = self.sample_v(h_k)    # sample visible from hidden
        return v_k  # k-step reconstructed visible

    def free_energy(self, v):
        # Compute the visible bias term for each sample in the batch
        vbias_term = (v * self.v_bias).sum(dim=1)  # shape: [batch_size]
        # Compute the activation of the hidden units
        wx_b = F.linear(v, self.W, self.h_bias)     # shape: [batch_size, n_hidden]
        # Compute the hidden term
        hidden_term = torch.sum(torch.log1p(torch.exp(wx_b)), dim=1)  # shape: [batch_size]
        # Return the mean free energy over the batch
        return - (vbias_term + hidden_term).mean()
    
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.FashionMNIST(root='./data', train=True, transform=transform, download=True)
test_dataset = datasets.FashionMNIST(root='./data', train=False, transform=transform, download=True)

def objective(trial):
    num_rbm_epochs = trial.suggest_int("num_rbm_epochs", 5, 5)# 24, 33)
    batch_size = trial.suggest_int("batch_size", 192, 1024)
    rbm_lr = trial.suggest_float("rbm_lr", 0.05, 0.1)
    rbm_hidden = trial.suggest_int("rbm_hidden", 384, 8192)

    mlflow.start_run(experiment_id=experiment.experiment_id)
    if CLASSIFIER != 'LogisticRegression':
        fnn_hidden = trial.suggest_int("fnn_hidden", 192, 384)
        fnn_lr = trial.suggest_float("fnn_lr", 0.0001, 0.0025)
        mlflow.log_param("fnn_hidden", fnn_hidden)
        mlflow.log_param("fnn_lr", fnn_lr)

    num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)# 40, 60)

    mlflow.log_param("num_rbm_epochs", num_rbm_epochs)
    mlflow.log_param("batch_size", batch_size)
    mlflow.log_param("rbm_lr", rbm_lr)
    mlflow.log_param("rbm_hidden", rbm_hidden)
    mlflow.log_param("num_classifier_epochs", num_classifier_epochs)

    # Instantiate RBM and optimizer
    device = torch.device("mps")
    rbm = RBM(n_visible=784, n_hidden=rbm_hidden, k=1).to(device)
    optimizer = torch.optim.SGD(rbm.parameters(), lr=rbm_lr)

    train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

    rbm_training_failed = False
    # Training loop (assuming train_loader yields batches of images and labels)
    for epoch in range(num_rbm_epochs):
        total_loss = 0.0
        for images, _ in train_loader:
            # Flatten images and binarize
            v0 = images.view(-1, 784).to(rbm.W.device)      # shape [batch_size, 784]
            v0 = torch.bernoulli(v0)                        # sample binary input
            vk = rbm(v0)                                    # k-step CD reconstruction
            # Compute contrastive divergence loss (free energy difference)
            loss = rbm.free_energy(v0) - rbm.free_energy(vk)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        print(f"Epoch {epoch+1}: avg free-energy loss = {total_loss/len(train_loader):.4f}")
        if np.isnan(total_loss):
            rbm_training_failed = True
            break

        if rbm_training_failed:
            accuracy = 0.0
            macro_f1 = 0.0 
            print("RBM training failed — returning 0.0 for accuracy and macro F1")  # 🔥 for visibility
            mlflow.log_metric("test_accuracy", accuracy)
            mlflow.log_metric("macro_f1", macro_f1)
            mlflow.set_tag("status", "rbm_failed")  # Optional tag
            mlflow.end_run()
            return float(accuracy)
    else:
        rbm.eval()  # set in evaluation mode if using any layers that behave differently in training
        features_list = []
        labels_list = []
        for images, labels in train_loader:
            v = images.view(-1, 784).to(rbm.W.device)
            v = v  # (optionally binarize or use raw normalized pixels)
            h_prob, h_sample = rbm.sample_h(v)  # get hidden activations
            features_list.append(h_prob.cpu().detach().numpy())
            labels_list.append(labels.numpy())
        train_features = np.concatenate(features_list)  # shape: [N_train, n_hidden]
        train_labels = np.concatenate(labels_list)

        # Convert pre-extracted training features and labels to tensors and create a DataLoader
        train_features_tensor = torch.tensor(train_features, dtype=torch.float32)
        train_labels_tensor = torch.tensor(train_labels, dtype=torch.long)
        train_feature_dataset = torch.utils.data.TensorDataset(train_features_tensor, train_labels_tensor)
        train_feature_loader = torch.utils.data.DataLoader(train_feature_dataset, batch_size=batch_size, shuffle=True)

            
        if CLASSIFIER == 'LogisticRegression':
            # add optuna tuning same as log reg without RBM features...
            lr_C = trial.suggest_float("lr_C", 0.01, 10.0, log=True)  
            mlflow.log_param("lr_C", lr_C)  # Log the chosen C value

            classifier = LogisticRegression(max_iter=num_classifier_epochs, C=lr_C, solver="saga") 
            classifier.fit(train_features, train_labels)            
            
        else:
            classifier = nn.Sequential(
                nn.Linear(rbm.n_hidden, fnn_hidden),
                nn.ReLU(),
                nn.Linear(fnn_hidden, 10)
            )

            # Move classifier to the same device as the RBM
            classifier = classifier.to(device)
            criterion = nn.CrossEntropyLoss()
            classifier_optimizer = torch.optim.Adam(classifier.parameters(), lr=fnn_lr)

            classifier.train()
            for epoch in range(num_classifier_epochs):
                running_loss = 0.0
                for features, labels in train_feature_loader:
                    features = features.to(device)
                    labels = labels.to(device)
                    
                    # Forward pass through classifier
                    outputs = classifier(features)
                    loss = criterion(outputs, labels)
                    
                    # Backpropagation and optimization
                    classifier_optimizer.zero_grad()
                    loss.backward()
                    classifier_optimizer.step()
                    
                    running_loss += loss.item()
                avg_loss = running_loss / len(train_feature_loader)
                print(f"Classifier Epoch {epoch+1}: loss = {avg_loss:.4f}")

        # Evaluate the classifier on test data.
        # Here we extract features from the RBM for each test image.
        if CLASSIFIER != 'LogisticRegression':
            classifier.eval()
            correct = 0
            total = 0
        features_list = []
        labels_list = []
        with torch.no_grad():
            for images, labels in test_loader:
                v = images.view(-1, 784).to(device)
                # Extract hidden activations; you can use either h_prob or h_sample.
                h_prob, _ = rbm.sample_h(v)
                if CLASSIFIER == 'LogisticRegression':
                    features_list.append(h_prob.cpu().detach().numpy())
                    labels_list.append(labels.numpy())
                else:
                    outputs = classifier(h_prob)
                    _, predicted = torch.max(outputs.data, 1)
                    total += labels.size(0)
                    correct += (predicted.cpu() == labels).sum().item()

        if CLASSIFIER == 'LogisticRegression':
            test_features = np.concatenate(features_list)
            test_labels = np.concatenate(labels_list)
            predictions = classifier.predict(test_features)
            accuracy = accuracy_score(test_labels, predictions) * 100
        
            macro_f1 = f1_score(test_labels, predictions, average="macro") 
        
        else:
            accuracy = 100 * correct / total
        
            all_preds = [] 
            all_labels = [] 
            classifier.eval()
            with torch.no_grad():
                for images, labels in test_loader:
                    v = images.view(-1, 784).to(device)
                    h_prob, _ = rbm.sample_h(v)
                    outputs = classifier(h_prob)
                    _, predicted = torch.max(outputs.data, 1)
                    all_preds.extend(predicted.cpu().numpy()) 
                    all_labels.extend(labels.numpy()) 
        
            macro_f1 = f1_score(all_labels, all_preds, average="macro") 
        
        print(f"Test Accuracy: {accuracy:.2f}%")
        print(f"Macro F1 Score: {macro_f1:.4f}") 
        
        mlflow.log_metric("test_accuracy", accuracy)
        mlflow.log_metric("macro_f1", macro_f1) 
        mlflow.end_run()
        return float(accuracy if accuracy is not None else 0.0)

if __name__ == "__main__":
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=50)
    print(study.best_params)
    print(study.best_value)
    print(study.best_trial)

Epoch 1: avg free-energy loss = 196.3455
Epoch 2: avg free-energy loss = 57.6702
Epoch 3: avg free-energy loss = 38.4611
Epoch 4: avg free-energy loss = 29.1952
Epoch 5: avg free-energy loss = 23.7323
Test Accuracy: 86.70%
Macro F1 Score: 0.8661
Epoch 1: avg free-energy loss = 193.4051
Epoch 2: avg free-energy loss = 56.0932
Epoch 3: avg free-energy loss = 38.1652
Epoch 4: avg free-energy loss = 29.3080
Epoch 5: avg free-energy loss = 23.3318
Test Accuracy: 85.46%
Macro F1 Score: 0.8536
Epoch 1: avg free-energy loss = 68.6174
Epoch 2: avg free-energy loss = 18.3964
Epoch 3: avg free-energy loss = 12.4841
Epoch 4: avg free-energy loss = 9.9436
Epoch 5: avg free-energy loss = 8.4359
Test Accuracy: 85.89%
Macro F1 Score: 0.8572
Epoch 1: avg free-energy loss = 153.2774
Epoch 2: avg free-energy loss = 45.3368
Epoch 3: avg free-energy loss = 31.0316
Epoch 4: avg free-energy loss = 24.8075
Epoch 5: avg free-energy loss = 20.9564
Test Accuracy: 86.49%
Macro F1 Score: 0.8631
Epoch 1: avg free-energy loss = 5.0666
Epoch 2: avg free-energy loss = -2.4851
Epoch 3: avg free-energy loss = -2.6238
Epoch 4: avg free-energy loss = -2.6816
Epoch 5: avg free-energy loss = -2.3391
Test Accuracy: 85.80%
Macro F1 Score: 0.8564
Epoch 1: avg free-energy loss = 172.9010
Epoch 2: avg free-energy loss = 38.1210
Epoch 3: avg free-energy loss = 26.1850
Epoch 4: avg free-energy loss = 20.3122
Epoch 5: avg free-energy loss = 17.5632
Test Accuracy: 86.32%
Macro F1 Score: 0.8626
Epoch 1: avg free-energy loss = 11.2064
Epoch 2: avg free-energy loss = -4.4292
Epoch 3: avg free-energy loss = -4.5026
Epoch 4: avg free-energy loss = -4.8340
Epoch 5: avg free-energy loss = -4.8850
Test Accuracy: 85.29%
Macro F1 Score: 0.8517
Epoch 1: avg free-energy loss = -9.3893
Epoch 2: avg free-energy loss = -13.4576
Epoch 3: avg free-energy loss = -11.4321
Epoch 4: avg free-energy loss = -10.5042
Epoch 5: avg free-energy loss = -9.7352
Test Accuracy: 85.17%
Macro F1 Score: 0.8506
Epoch 1: avg free-energy loss = 9.0801
Epoch 2: avg free-energy loss = 0.1499
Epoch 3: avg free-energy loss = 0.1757
Epoch 4: avg free-energy loss = -0.0278
Epoch 5: avg free-energy loss = 0.0360
Test Accuracy: 86.10%
Macro F1 Score: 0.8605
Epoch 1: avg free-energy loss = 46.4491
Epoch 2: avg free-energy loss = 12.1879
Epoch 3: avg free-energy loss = 9.0373
Epoch 4: avg free-energy loss = 7.4208
Epoch 5: avg free-energy loss = 6.5886
Test Accuracy: 86.51%
Macro F1 Score: 0.8644
Epoch 1: avg free-energy loss = 458.1128
Epoch 2: avg free-energy loss = 132.8139
Epoch 3: avg free-energy loss = 86.0433
Epoch 4: avg free-energy loss = 67.3414
Epoch 5: avg free-energy loss = 55.3441
Test Accuracy: 86.51%
Macro F1 Score: 0.8644
Epoch 1: avg free-energy loss = 101.0917
Epoch 2: avg free-energy loss = 27.5174
Epoch 3: avg free-energy loss = 17.7405
Epoch 4: avg free-energy loss = 13.9116
Epoch 5: avg free-energy loss = 11.9980
Test Accuracy: 86.19%
Macro F1 Score: 0.8619
Epoch 1: avg free-energy loss = 64.8062
Epoch 2: avg free-energy loss = 22.3084
Epoch 3: avg free-energy loss = 17.7644
Epoch 4: avg free-energy loss = 15.4338
Epoch 5: avg free-energy loss = 13.7978
Test Accuracy: 86.75%
Macro F1 Score: 0.8667
Epoch 1: avg free-energy loss = 78.5650
Epoch 2: avg free-energy loss = 26.2359
Epoch 3: avg free-energy loss = 20.0685
Epoch 4: avg free-energy loss = 16.9519
Epoch 5: avg free-energy loss = 15.0379
Test Accuracy: 86.67%
Macro F1 Score: 0.8658
Epoch 1: avg free-energy loss = 350.3286
Epoch 2: avg free-energy loss = 108.6761
Epoch 3: avg free-energy loss = 72.9799
Epoch 4: avg free-energy loss = 55.5765
Epoch 5: avg free-energy loss = 44.3201
Test Accuracy: 86.37%
Macro F1 Score: 0.8623
Epoch 1: avg free-energy loss = 107.4147
Epoch 2: avg free-energy loss = 36.1246
Epoch 3: avg free-energy loss = 27.0512
Epoch 4: avg free-energy loss = 22.4735
Epoch 5: avg free-energy loss = 18.8480
Test Accuracy: 86.78%
Macro F1 Score: 0.8667
Epoch 1: avg free-energy loss = 62.1499
Epoch 2: avg free-energy loss = 20.3970
Epoch 3: avg free-energy loss = 15.6454
Epoch 4: avg free-energy loss = 13.4197
Epoch 5: avg free-energy loss = 11.8039
Test Accuracy: 86.82%
Macro F1 Score: 0.8676
Epoch 1: avg free-energy loss = 210.0479
Epoch 2: avg free-energy loss = 61.5162
Epoch 3: avg free-energy loss = 40.8242
Epoch 4: avg free-energy loss = 32.0612
Epoch 5: avg free-energy loss = 26.5049
Test Accuracy: 86.56%
Macro F1 Score: 0.8650
Epoch 1: avg free-energy loss = 53.0006
Epoch 2: avg free-energy loss = 17.9123
Epoch 3: avg free-energy loss = 14.0345
Epoch 4: avg free-energy loss = 12.1547
Epoch 5: avg free-energy loss = 10.7802
Test Accuracy: 86.39%
Macro F1 Score: 0.8635
Epoch 1: avg free-energy loss = 139.4416
Epoch 2: avg free-energy loss = 39.5729
Epoch 3: avg free-energy loss = 27.4402
Epoch 4: avg free-energy loss = 21.8295
Epoch 5: avg free-energy loss = 18.7981
Test Accuracy: 86.46%
Macro F1 Score: 0.8639
Epoch 1: avg free-energy loss = 63.5245
Epoch 2: avg free-energy loss = 15.8937
Epoch 3: avg free-energy loss = 10.5403
Epoch 4: avg free-energy loss = 8.2907
Epoch 5: avg free-energy loss = 6.9727
Test Accuracy: 86.52%
Macro F1 Score: 0.8645
Epoch 1: avg free-energy loss = 69.5283
Epoch 2: avg free-energy loss = 22.5561
Epoch 3: avg free-energy loss = 17.4453
Epoch 4: avg free-energy loss = 14.9944
Epoch 5: avg free-energy loss = 13.4487
Test Accuracy: 86.69%
Macro F1 Score: 0.8662
Epoch 1: avg free-energy loss = 65.5971
Epoch 2: avg free-energy loss = 23.8052
Epoch 3: avg free-energy loss = 19.6401
Epoch 4: avg free-energy loss = 17.0054
Epoch 5: avg free-energy loss = 15.6487
Test Accuracy: 86.56%
Macro F1 Score: 0.8646
Epoch 1: avg free-energy loss = 97.8708
Epoch 2: avg free-energy loss = 26.8091
Epoch 3: avg free-energy loss = 19.4019
Epoch 4: avg free-energy loss = 15.8896
Epoch 5: avg free-energy loss = 13.8290
Test Accuracy: 86.46%
Macro F1 Score: 0.8639
Epoch 1: avg free-energy loss = 62.3792
Epoch 2: avg free-energy loss = 18.5537
Epoch 3: avg free-energy loss = 13.8053
Epoch 4: avg free-energy loss = 11.6182
Epoch 5: avg free-energy loss = 10.1345
Test Accuracy: 86.41%
Macro F1 Score: 0.8624
Epoch 1: avg free-energy loss = 140.3730
Epoch 2: avg free-energy loss = 40.8663
Epoch 3: avg free-energy loss = 28.1880
Epoch 4: avg free-energy loss = 22.3220
Epoch 5: avg free-energy loss = 18.8671
Test Accuracy: 86.72%
Macro F1 Score: 0.8661
Epoch 1: avg free-energy loss = 35.9828
Epoch 2: avg free-energy loss = 9.1321
Epoch 3: avg free-energy loss = 6.1407
Epoch 4: avg free-energy loss = 4.7699
Epoch 5: avg free-energy loss = 4.0476
Test Accuracy: 86.65%
Macro F1 Score: 0.8655
Epoch 1: avg free-energy loss = 53.7467
Epoch 2: avg free-energy loss = 18.0739
Epoch 3: avg free-energy loss = 14.2344
Epoch 4: avg free-energy loss = 12.3961
Epoch 5: avg free-energy loss = 11.0257
Test Accuracy: 86.56%
Macro F1 Score: 0.8648
Epoch 1: avg free-energy loss = 188.3934
Epoch 2: avg free-energy loss = 54.3442
Epoch 3: avg free-energy loss = 36.2564
Epoch 4: avg free-energy loss = 27.5151
Epoch 5: avg free-energy loss = 22.8576
Test Accuracy: 86.58%
Macro F1 Score: 0.8651
Epoch 1: avg free-energy loss = 113.5767
Epoch 2: avg free-energy loss = 35.7491
Epoch 3: avg free-energy loss = 25.8849
Epoch 4: avg free-energy loss = 21.0765
Epoch 5: avg free-energy loss = 18.4651
Test Accuracy: 86.65%
Macro F1 Score: 0.8655
Epoch 1: avg free-energy loss = 200.1413
Epoch 2: avg free-energy loss = 55.9235
Epoch 3: avg free-energy loss = 35.9069
Epoch 4: avg free-energy loss = 29.4727
Epoch 5: avg free-energy loss = 24.1111
Test Accuracy: 86.25%
Macro F1 Score: 0.8619
Epoch 1: avg free-energy loss = 126.6828
Epoch 2: avg free-energy loss = 36.2785
Epoch 3: avg free-energy loss = 25.5911
Epoch 4: avg free-energy loss = 20.6815
Epoch 5: avg free-energy loss = 17.9402
Test Accuracy: 86.43%
Macro F1 Score: 0.8635
Epoch 1: avg free-energy loss = 112.7345
Epoch 2: avg free-energy loss = 33.4477
Epoch 3: avg free-energy loss = 23.7942
Epoch 4: avg free-energy loss = 19.1399
Epoch 5: avg free-energy loss = 16.3007
Test Accuracy: 86.74%
Macro F1 Score: 0.8671
Epoch 1: avg free-energy loss = 105.6741
Epoch 2: avg free-energy loss = 31.2973
Epoch 3: avg free-energy loss = 22.2810
Epoch 4: avg free-energy loss = 17.9960
Epoch 5: avg free-energy loss = 15.4486
Test Accuracy: 86.67%
Macro F1 Score: 0.8660
Epoch 1: avg free-energy loss = 88.4828
Epoch 2: avg free-energy loss = 24.1312
Epoch 3: avg free-energy loss = 16.8558
Epoch 4: avg free-energy loss = 13.4285
Epoch 5: avg free-energy loss = 11.2438
Test Accuracy: 86.05%
Macro F1 Score: 0.8595
Epoch 1: avg free-energy loss = 164.9203
Epoch 2: avg free-energy loss = 47.3598
Epoch 3: avg free-energy loss = 30.9522
Epoch 4: avg free-energy loss = 24.6439
Epoch 5: avg free-energy loss = 20.8669
Test Accuracy: 86.51%
Macro F1 Score: 0.8649
Epoch 1: avg free-energy loss = 115.2898
Epoch 2: avg free-energy loss = 36.4158
Epoch 3: avg free-energy loss = 26.5679
Epoch 4: avg free-energy loss = 20.0425
Epoch 5: avg free-energy loss = 18.5737
Test Accuracy: 86.55%
Macro F1 Score: 0.8645
Epoch 1: avg free-energy loss = 131.9440
Epoch 2: avg free-energy loss = 34.4924
Epoch 3: avg free-energy loss = 25.0261
Epoch 4: avg free-energy loss = 20.8463
Epoch 5: avg free-energy loss = 18.3044
Test Accuracy: 86.39%
Macro F1 Score: 0.8630
Epoch 1: avg free-energy loss = 52.8054
Epoch 2: avg free-energy loss = 14.8391
Epoch 3: avg free-energy loss = 11.0946
Epoch 4: avg free-energy loss = 9.2083
Epoch 5: avg free-energy loss = 8.0666
Test Accuracy: 86.34%
Macro F1 Score: 0.8625
Epoch 1: avg free-energy loss = 129.3992
Epoch 2: avg free-energy loss = 36.5567
Epoch 3: avg free-energy loss = 24.5337
Epoch 4: avg free-energy loss = 19.3778
Epoch 5: avg free-energy loss = 16.1646
Test Accuracy: 85.32%
Macro F1 Score: 0.8520
Epoch 1: avg free-energy loss = 249.2433
Epoch 2: avg free-energy loss = 77.2602
Epoch 3: avg free-energy loss = 51.5478
Epoch 4: avg free-energy loss = 40.2061
Epoch 5: avg free-energy loss = 31.2089
Test Accuracy: 86.31%
Macro F1 Score: 0.8619
Epoch 1: avg free-energy loss = 151.3303
Epoch 2: avg free-energy loss = 44.6183
Epoch 3: avg free-energy loss = 30.0300
Epoch 4: avg free-energy loss = 23.6704
Epoch 5: avg free-energy loss = 19.9832
Test Accuracy: 86.68%
Macro F1 Score: 0.8657
Epoch 1: avg free-energy loss = 110.8060
Epoch 2: avg free-energy loss = 32.9394
Epoch 3: avg free-energy loss = 24.3679
Epoch 4: avg free-energy loss = 20.0207
Epoch 5: avg free-energy loss = 17.7401
Test Accuracy: 86.55%
Macro F1 Score: 0.8654
Epoch 1: avg free-energy loss = 151.8053
Epoch 2: avg free-energy loss = 42.8811
Epoch 3: avg free-energy loss = 29.1478
Epoch 4: avg free-energy loss = 23.6812
Epoch 5: avg free-energy loss = 20.2689
Test Accuracy: 86.23%
Macro F1 Score: 0.8631
Epoch 1: avg free-energy loss = 71.3464
Epoch 2: avg free-energy loss = 24.3363
Epoch 3: avg free-energy loss = 19.0738
Epoch 4: avg free-energy loss = 16.3010
Epoch 5: avg free-energy loss = 14.7047
Test Accuracy: 86.37%
Macro F1 Score: 0.8625
Epoch 1: avg free-energy loss = 58.6644
Epoch 2: avg free-energy loss = 17.0111
Epoch 3: avg free-energy loss = 12.2987
Epoch 4: avg free-energy loss = 10.0023
Epoch 5: avg free-energy loss = 8.6010
Test Accuracy: 86.76%
Macro F1 Score: 0.8666
Epoch 1: avg free-energy loss = 34.5321
Epoch 2: avg free-energy loss = 8.6056
Epoch 3: avg free-energy loss = 5.7289
Epoch 4: avg free-energy loss = 4.6159
Epoch 5: avg free-energy loss = 3.8419
Test Accuracy: 86.56%
Macro F1 Score: 0.8646
Epoch 1: avg free-energy loss = 59.3033
Epoch 2: avg free-energy loss = 17.5087
Epoch 3: avg free-energy loss = 12.4595
Epoch 4: avg free-energy loss = 10.1173
Epoch 5: avg free-energy loss = 8.6744
Test Accuracy: 86.79%
Macro F1 Score: 0.8666
Epoch 1: avg free-energy loss = 90.6978
Epoch 2: avg free-energy loss = 24.3815
Epoch 3: avg free-energy loss = 15.3271
Epoch 4: avg free-energy loss = 11.7499
Epoch 5: avg free-energy loss = 9.5954
Test Accuracy: 86.43%
Macro F1 Score: 0.8634
Epoch 1: avg free-energy loss = 126.1584
Epoch 2: avg free-energy loss = 36.4904
Epoch 3: avg free-energy loss = 23.2756
Epoch 4: avg free-energy loss = 17.2837
Epoch 5: avg free-energy loss = 14.0218
Test Accuracy: 86.38%
Macro F1 Score: 0.8626
{'num_rbm_epochs': 5, 'batch_size': 219, 'rbm_lr': 0.07193606666763605, 'rbm_hidden': 3517, 'num_classifier_epochs': 5, 'lr_C': 2.8856789323424294}
86.82
FrozenTrial(number=16, state=1, values=[86.82], datetime_start=datetime.datetime(2025, 3, 28, 15, 58, 3, 875747), datetime_complete=datetime.datetime(2025, 3, 28, 15, 58, 56, 977521), params={'num_rbm_epochs': 5, 'batch_size': 219, 'rbm_lr': 0.07193606666763605, 'rbm_hidden': 3517, 'num_classifier_epochs': 5, 'lr_C': 2.8856789323424294}, user_attrs={}, system_attrs={}, intermediate_values={}, distributions={'num_rbm_epochs': IntDistribution(high=5, log=False, low=5, step=1), 'batch_size': IntDistribution(high=1024, log=False, low=192, step=1), 'rbm_lr': FloatDistribution(high=0.1, log=False, low=0.05, step=None), 'rbm_hidden': IntDistribution(high=8192, log=False, low=384, step=1), 'num_classifier_epochs': IntDistribution(high=5, log=False, low=5, step=1), 'lr_C': FloatDistribution(high=10.0, log=True, low=0.01, step=None)}, trial_id=16, value=None)

[I 2025-03-28 15:43:28,289] A new study created in memory with name: no-name-d2c94975-041e-47d3-8353-8fdc1f881cf7
[I 2025-03-28 15:44:36,278] Trial 0 finished with value: 86.7 and parameters: {'num_rbm_epochs': 5, 'batch_size': 475, 'rbm_lr': 0.08471941123761287, 'rbm_hidden': 5360, 'num_classifier_epochs': 5, 'lr_C': 7.759560632006714}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:45:37,062] Trial 1 finished with value: 85.46000000000001 and parameters: {'num_rbm_epochs': 5, 'batch_size': 581, 'rbm_lr': 0.08580593423214694, 'rbm_hidden': 4840, 'num_classifier_epochs': 5, 'lr_C': 0.010436675317281851}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:46:17,070] Trial 2 finished with value: 85.89 and parameters: {'num_rbm_epochs': 5, 'batch_size': 480, 'rbm_lr': 0.07869425006404021, 'rbm_hidden': 2736, 'num_classifier_epochs': 5, 'lr_C': 0.023336014276379}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:47:35,007] Trial 3 finished with value: 86.49 and parameters: {'num_rbm_epochs': 5, 'batch_size': 270, 'rbm_lr': 0.07800623930294098, 'rbm_hidden': 5940, 'num_classifier_epochs': 5, 'lr_C': 6.868972291798769}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:48:02,771] Trial 4 finished with value: 85.8 and parameters: {'num_rbm_epochs': 5, 'batch_size': 323, 'rbm_lr': 0.09497212684905881, 'rbm_hidden': 1227, 'num_classifier_epochs': 5, 'lr_C': 2.053826787090791}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:49:24,147] Trial 5 finished with value: 86.32 and parameters: {'num_rbm_epochs': 5, 'batch_size': 499, 'rbm_lr': 0.059483581764094705, 'rbm_hidden': 6633, 'num_classifier_epochs': 5, 'lr_C': 0.8327307282767233}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:49:48,728] Trial 6 finished with value: 85.28999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 830, 'rbm_lr': 0.0947445517768846, 'rbm_hidden': 1174, 'num_classifier_epochs': 5, 'lr_C': 0.08910458198589148}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:50:11,644] Trial 7 finished with value: 85.17 and parameters: {'num_rbm_epochs': 5, 'batch_size': 573, 'rbm_lr': 0.0622047877721967, 'rbm_hidden': 866, 'num_classifier_epochs': 5, 'lr_C': 3.8974699887946844}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:50:41,435] Trial 8 finished with value: 86.1 and parameters: {'num_rbm_epochs': 5, 'batch_size': 396, 'rbm_lr': 0.05280664101970556, 'rbm_hidden': 1527, 'num_classifier_epochs': 5, 'lr_C': 2.0938617043310113}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:51:21,041] Trial 9 finished with value: 86.50999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 486, 'rbm_lr': 0.06369736391427752, 'rbm_hidden': 2425, 'num_classifier_epochs': 5, 'lr_C': 0.9812304011844255}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:52:56,029] Trial 10 finished with value: 86.50999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 797, 'rbm_lr': 0.0862451943680599, 'rbm_hidden': 8148, 'num_classifier_epochs': 5, 'lr_C': 0.19658554788603524}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:53:40,293] Trial 11 finished with value: 86.19 and parameters: {'num_rbm_epochs': 5, 'batch_size': 718, 'rbm_lr': 0.06807577425840475, 'rbm_hidden': 3160, 'num_classifier_epochs': 5, 'lr_C': 0.6822972143991152}. Best is trial 0 with value: 86.7.
[I 2025-03-28 15:54:37,900] Trial 12 finished with value: 86.75 and parameters: {'num_rbm_epochs': 5, 'batch_size': 197, 'rbm_lr': 0.0672593918743455, 'rbm_hidden': 3891, 'num_classifier_epochs': 5, 'lr_C': 8.784761364782293}. Best is trial 12 with value: 86.75.
[I 2025-03-28 15:55:39,902] Trial 13 finished with value: 86.67 and parameters: {'num_rbm_epochs': 5, 'batch_size': 192, 'rbm_lr': 0.07151373619461751, 'rbm_hidden': 4303, 'num_classifier_epochs': 5, 'lr_C': 8.37571699195763}. Best is trial 12 with value: 86.75.
[I 2025-03-28 15:56:52,177] Trial 14 finished with value: 86.37 and parameters: {'num_rbm_epochs': 5, 'batch_size': 983, 'rbm_lr': 0.087013841826266, 'rbm_hidden': 5905, 'num_classifier_epochs': 5, 'lr_C': 9.802078153234888}. Best is trial 12 with value: 86.75.
[I 2025-03-28 15:58:03,875] Trial 15 finished with value: 86.78 and parameters: {'num_rbm_epochs': 5, 'batch_size': 212, 'rbm_lr': 0.07495019459233111, 'rbm_hidden': 5066, 'num_classifier_epochs': 5, 'lr_C': 3.493191411482752}. Best is trial 15 with value: 86.78.
[I 2025-03-28 15:58:56,977] Trial 16 finished with value: 86.82 and parameters: {'num_rbm_epochs': 5, 'batch_size': 219, 'rbm_lr': 0.07193606666763605, 'rbm_hidden': 3517, 'num_classifier_epochs': 5, 'lr_C': 2.8856789323424294}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:00:27,317] Trial 17 finished with value: 86.56 and parameters: {'num_rbm_epochs': 5, 'batch_size': 337, 'rbm_lr': 0.0751750310772006, 'rbm_hidden': 7173, 'num_classifier_epochs': 5, 'lr_C': 0.2554998104507218}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:01:20,557] Trial 18 finished with value: 86.39 and parameters: {'num_rbm_epochs': 5, 'batch_size': 267, 'rbm_lr': 0.05113390269488412, 'rbm_hidden': 3637, 'num_classifier_epochs': 5, 'lr_C': 2.31371612518345}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:02:23,985] Trial 19 finished with value: 86.46000000000001 and parameters: {'num_rbm_epochs': 5, 'batch_size': 362, 'rbm_lr': 0.07924887131431342, 'rbm_hidden': 4786, 'num_classifier_epochs': 5, 'lr_C': 0.5476362288752789}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:02:59,789] Trial 20 finished with value: 86.52 and parameters: {'num_rbm_epochs': 5, 'batch_size': 694, 'rbm_lr': 0.072089240302831, 'rbm_hidden': 2353, 'num_classifier_epochs': 5, 'lr_C': 3.120325582059925}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:03:56,094] Trial 21 finished with value: 86.69 and parameters: {'num_rbm_epochs': 5, 'batch_size': 226, 'rbm_lr': 0.06717784476562857, 'rbm_hidden': 3876, 'num_classifier_epochs': 5, 'lr_C': 3.5407651486029876}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:04:58,762] Trial 22 finished with value: 86.56 and parameters: {'num_rbm_epochs': 5, 'batch_size': 198, 'rbm_lr': 0.058106399403143175, 'rbm_hidden': 4369, 'num_classifier_epochs': 5, 'lr_C': 1.7095805800511936}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:05:52,143] Trial 23 finished with value: 86.46000000000001 and parameters: {'num_rbm_epochs': 5, 'batch_size': 405, 'rbm_lr': 0.06766068429026827, 'rbm_hidden': 3885, 'num_classifier_epochs': 5, 'lr_C': 4.477028191264163}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:06:40,208] Trial 24 finished with value: 86.41 and parameters: {'num_rbm_epochs': 5, 'batch_size': 278, 'rbm_lr': 0.07253434768410301, 'rbm_hidden': 3158, 'num_classifier_epochs': 5, 'lr_C': 1.2511154873387287}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:07:49,657] Trial 25 finished with value: 86.72 and parameters: {'num_rbm_epochs': 5, 'batch_size': 301, 'rbm_lr': 0.08143637749521292, 'rbm_hidden': 5153, 'num_classifier_epochs': 5, 'lr_C': 4.583263918634515}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:08:24,552] Trial 26 finished with value: 86.65 and parameters: {'num_rbm_epochs': 5, 'batch_size': 421, 'rbm_lr': 0.07491001982477367, 'rbm_hidden': 2045, 'num_classifier_epochs': 5, 'lr_C': 5.198087737600761}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:09:14,521] Trial 27 finished with value: 86.56 and parameters: {'num_rbm_epochs': 5, 'batch_size': 244, 'rbm_lr': 0.0661987651359746, 'rbm_hidden': 3261, 'num_classifier_epochs': 5, 'lr_C': 0.37398397167067693}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:10:29,913] Trial 28 finished with value: 86.58 and parameters: {'num_rbm_epochs': 5, 'batch_size': 346, 'rbm_lr': 0.09081361850775344, 'rbm_hidden': 5786, 'num_classifier_epochs': 5, 'lr_C': 1.264485858104714}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:11:43,825] Trial 29 finished with value: 86.65 and parameters: {'num_rbm_epochs': 5, 'batch_size': 195, 'rbm_lr': 0.08194330768864505, 'rbm_hidden': 5315, 'num_classifier_epochs': 5, 'lr_C': 0.11351196121705188}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:13:06,107] Trial 30 finished with value: 86.25 and parameters: {'num_rbm_epochs': 5, 'batch_size': 419, 'rbm_lr': 0.07010029879962065, 'rbm_hidden': 6580, 'num_classifier_epochs': 5, 'lr_C': 6.435433746499978}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:14:11,022] Trial 31 finished with value: 86.42999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 308, 'rbm_lr': 0.07572303417240833, 'rbm_hidden': 4930, 'num_classifier_epochs': 5, 'lr_C': 9.838619339807405}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:15:13,341] Trial 32 finished with value: 86.74 and parameters: {'num_rbm_epochs': 5, 'batch_size': 275, 'rbm_lr': 0.08197454239219443, 'rbm_hidden': 4534, 'num_classifier_epochs': 5, 'lr_C': 3.0717474749785127}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:16:16,544] Trial 33 finished with value: 86.67 and parameters: {'num_rbm_epochs': 5, 'batch_size': 247, 'rbm_lr': 0.08287714864162257, 'rbm_hidden': 4530, 'num_classifier_epochs': 5, 'lr_C': 3.0461369518294736}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:17:13,254] Trial 34 finished with value: 86.05000000000001 and parameters: {'num_rbm_epochs': 5, 'batch_size': 241, 'rbm_lr': 0.09905318431029067, 'rbm_hidden': 3850, 'num_classifier_epochs': 5, 'lr_C': 0.03650522628066981}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:18:24,206] Trial 35 finished with value: 86.50999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 363, 'rbm_lr': 0.07828664650479517, 'rbm_hidden': 5509, 'num_classifier_epochs': 5, 'lr_C': 1.4839657304578404}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:19:12,345] Trial 36 finished with value: 86.55000000000001 and parameters: {'num_rbm_epochs': 5, 'batch_size': 566, 'rbm_lr': 0.0887882829697821, 'rbm_hidden': 3411, 'num_classifier_epochs': 5, 'lr_C': 6.0363163478042186}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:20:34,442] Trial 37 finished with value: 86.39 and parameters: {'num_rbm_epochs': 5, 'batch_size': 294, 'rbm_lr': 0.06250176363247048, 'rbm_hidden': 6433, 'num_classifier_epochs': 5, 'lr_C': 2.597487073433132}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:21:14,268] Trial 38 finished with value: 86.33999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 521, 'rbm_lr': 0.05865245219431822, 'rbm_hidden': 2634, 'num_classifier_epochs': 5, 'lr_C': 6.512258803354044}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:22:10,136] Trial 39 finished with value: 85.32 and parameters: {'num_rbm_epochs': 5, 'batch_size': 459, 'rbm_lr': 0.07654689776619607, 'rbm_hidden': 4189, 'num_classifier_epochs': 5, 'lr_C': 0.0108980040443134}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:23:09,972] Trial 40 finished with value: 86.31 and parameters: {'num_rbm_epochs': 5, 'batch_size': 974, 'rbm_lr': 0.08010571895281741, 'rbm_hidden': 4751, 'num_classifier_epochs': 5, 'lr_C': 0.5058272457049481}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:24:20,722] Trial 41 finished with value: 86.68 and parameters: {'num_rbm_epochs': 5, 'batch_size': 296, 'rbm_lr': 0.08277002363450732, 'rbm_hidden': 5443, 'num_classifier_epochs': 5, 'lr_C': 4.642558835957204}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:25:29,834] Trial 42 finished with value: 86.55000000000001 and parameters: {'num_rbm_epochs': 5, 'batch_size': 237, 'rbm_lr': 0.07424871127913, 'rbm_hidden': 5063, 'num_classifier_epochs': 5, 'lr_C': 3.843158063929651}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:26:48,465] Trial 43 finished with value: 86.22999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 300, 'rbm_lr': 0.06956126534306944, 'rbm_hidden': 6206, 'num_classifier_epochs': 5, 'lr_C': 1.9318458526470277}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:27:49,361] Trial 44 finished with value: 86.37 and parameters: {'num_rbm_epochs': 5, 'batch_size': 215, 'rbm_lr': 0.06488286400936794, 'rbm_hidden': 4162, 'num_classifier_epochs': 5, 'lr_C': 7.322715230877536}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:28:35,358] Trial 45 finished with value: 86.76 and parameters: {'num_rbm_epochs': 5, 'batch_size': 270, 'rbm_lr': 0.08567443169731266, 'rbm_hidden': 2947, 'num_classifier_epochs': 5, 'lr_C': 1.0407917550368337}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:29:10,580] Trial 46 finished with value: 86.56 and parameters: {'num_rbm_epochs': 5, 'batch_size': 373, 'rbm_lr': 0.08484078082719072, 'rbm_hidden': 2031, 'num_classifier_epochs': 5, 'lr_C': 1.0235821758061208}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:29:56,739] Trial 47 finished with value: 86.79 and parameters: {'num_rbm_epochs': 5, 'batch_size': 272, 'rbm_lr': 0.08873858134221667, 'rbm_hidden': 2942, 'num_classifier_epochs': 5, 'lr_C': 0.8053881598695234}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:30:38,118] Trial 48 finished with value: 86.42999999999999 and parameters: {'num_rbm_epochs': 5, 'batch_size': 642, 'rbm_lr': 0.09361630741085752, 'rbm_hidden': 2785, 'num_classifier_epochs': 5, 'lr_C': 0.6023133401443783}. Best is trial 16 with value: 86.82.
[I 2025-03-28 16:31:20,831] Trial 49 finished with value: 86.38 and parameters: {'num_rbm_epochs': 5, 'batch_size': 895, 'rbm_lr': 0.09136768154625116, 'rbm_hidden': 2986, 'num_classifier_epochs': 5, 'lr_C': 0.8668116354798511}. Best is trial 16 with value: 86.82.

Test Accuracy of Logistic Regression on RBM Hidden Features by Inverse Regularization Strength

Test Accuracy By Number of RBM Hidden Units

Model 5: Feed Forward Network on RBM Hidden Features (of Fashion MNIST Data)

Click to Show Code and Output

from sklearn.metrics import accuracy_score, f1_score
CLASSIFIER = 'FNN'

if CLASSIFIER == 'LogisticRegression':
    experiment = mlflow.set_experiment("new-pytorch-fmnist-lr-withrbm")
else:
    experiment = mlflow.set_experiment("new-pytorch-fmnist-fnn-withrbm")


class RBM(nn.Module):
    def __init__(self, n_visible=784, n_hidden=256, k=1):
        super(RBM, self).__init__()
        self.n_visible = n_visible
        self.n_hidden = n_hidden
        # Initialize weights and biases
        self.W = nn.Parameter(torch.randn(n_hidden, n_visible) * 0.1)
        self.v_bias = nn.Parameter(torch.zeros(n_visible))
        self.h_bias = nn.Parameter(torch.zeros(n_hidden))
        self.k = k  # CD-k steps

    def sample_h(self, v):
        # Given visible v, sample hidden h
        p_h = torch.sigmoid(F.linear(v, self.W, self.h_bias))  # p(h=1|v)
        h_sample = torch.bernoulli(p_h)                        # sample Bernoulli
        return p_h, h_sample

    def sample_v(self, h):
        # Given hidden h, sample visible v
        p_v = torch.sigmoid(F.linear(h, self.W.t(), self.v_bias))  # p(v=1|h)
        v_sample = torch.bernoulli(p_v)
        return p_v, v_sample

    def forward(self, v):
        # Perform k steps of contrastive divergence starting from v
        v_k = v.clone()
        for _ in range(self.k):
            _, h_k = self.sample_h(v_k)    # sample hidden from current visible
            _, v_k = self.sample_v(h_k)    # sample visible from hidden
        return v_k  # k-step reconstructed visible

    def free_energy(self, v):
        # Compute the visible bias term for each sample in the batch
        vbias_term = (v * self.v_bias).sum(dim=1)  # shape: [batch_size]
        # Compute the activation of the hidden units
        wx_b = F.linear(v, self.W, self.h_bias)     # shape: [batch_size, n_hidden]
        # Compute the hidden term
        hidden_term = torch.sum(torch.log1p(torch.exp(wx_b)), dim=1)  # shape: [batch_size]
        # Return the mean free energy over the batch
        return - (vbias_term + hidden_term).mean()
    
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.FashionMNIST(root='./data', train=True, transform=transform, download=True)
test_dataset = datasets.FashionMNIST(root='./data', train=False, transform=transform, download=True)

def objective(trial):
    num_rbm_epochs = trial.suggest_int("num_rbm_epochs", 5, 5)# 24, 33)
    batch_size = trial.suggest_int("batch_size", 192, 1024)
    rbm_lr = trial.suggest_float("rbm_lr", 0.05, 0.1)
    rbm_hidden = trial.suggest_int("rbm_hidden", 384, 8192)

    mlflow.start_run(experiment_id=experiment.experiment_id)
    if CLASSIFIER != 'LogisticRegression':
        fnn_hidden = trial.suggest_int("fnn_hidden", 192, 384)
        fnn_lr = trial.suggest_float("fnn_lr", 0.0001, 0.0025)
        mlflow.log_param("fnn_hidden", fnn_hidden)
        mlflow.log_param("fnn_lr", fnn_lr)

    num_classifier_epochs = trial.suggest_int("num_classifier_epochs", 5, 5)# 40, 60)

    mlflow.log_param("num_rbm_epochs", num_rbm_epochs)
    mlflow.log_param("batch_size", batch_size)
    mlflow.log_param("rbm_lr", rbm_lr)
    mlflow.log_param("rbm_hidden", rbm_hidden)
    mlflow.log_param("num_classifier_epochs", num_classifier_epochs)

    # Instantiate RBM and optimizer
    device = torch.device("mps")
    rbm = RBM(n_visible=784, n_hidden=rbm_hidden, k=1).to(device)
    optimizer = torch.optim.SGD(rbm.parameters(), lr=rbm_lr)

    train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

    rbm_training_failed = False
    # Training loop (assuming train_loader yields batches of images and labels)
    for epoch in range(num_rbm_epochs):
        total_loss = 0.0
        for images, _ in train_loader:
            # Flatten images and binarize
            v0 = images.view(-1, 784).to(rbm.W.device)      # shape [batch_size, 784]
            v0 = torch.bernoulli(v0)                        # sample binary input
            vk = rbm(v0)                                    # k-step CD reconstruction
            # Compute contrastive divergence loss (free energy difference)
            loss = rbm.free_energy(v0) - rbm.free_energy(vk)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        print(f"Epoch {epoch+1}: avg free-energy loss = {total_loss/len(train_loader):.4f}")
        if np.isnan(total_loss):
            rbm_training_failed = True
            break

        if rbm_training_failed:
            accuracy = 0.0
            macro_f1 = 0.0 
            print("RBM training failed — returning 0.0 for accuracy and macro F1")  # 🔥 for visibility
            mlflow.log_metric("test_accuracy", accuracy)
            mlflow.log_metric("macro_f1", macro_f1)
            mlflow.set_tag("status", "rbm_failed")  # Optional tag
            mlflow.end_run()
            return float(accuracy)
    else:
        rbm.eval()  # set in evaluation mode if using any layers that behave differently in training
        features_list = []
        labels_list = []
        for images, labels in train_loader:
            v = images.view(-1, 784).to(rbm.W.device)
            v = v  # (optionally binarize or use raw normalized pixels)
            h_prob, h_sample = rbm.sample_h(v)  # get hidden activations
            features_list.append(h_prob.cpu().detach().numpy())
            labels_list.append(labels.numpy())
        train_features = np.concatenate(features_list)  # shape: [N_train, n_hidden]
        train_labels = np.concatenate(labels_list)

        # Convert pre-extracted training features and labels to tensors and create a DataLoader
        train_features_tensor = torch.tensor(train_features, dtype=torch.float32)
        train_labels_tensor = torch.tensor(train_labels, dtype=torch.long)
        train_feature_dataset = torch.utils.data.TensorDataset(train_features_tensor, train_labels_tensor)
        train_feature_loader = torch.utils.data.DataLoader(train_feature_dataset, batch_size=batch_size, shuffle=True)

            
        if CLASSIFIER == 'LogisticRegression':
            # add optuna tuning same as log reg without RBM features...
            lr_C = trial.suggest_float("lr_C", 0.01, 10.0, log=True)  
            mlflow.log_param("lr_C", lr_C)  # Log the chosen C value

            classifier = LogisticRegression(max_iter=num_classifier_epochs, C=lr_C, solver="saga") 
            classifier.fit(train_features, train_labels)            
            
        else:
            classifier = nn.Sequential(
                nn.Linear(rbm.n_hidden, fnn_hidden),
                nn.ReLU(),
                nn.Linear(fnn_hidden, 10)
            )

            # Move classifier to the same device as the RBM
            classifier = classifier.to(device)
            criterion = nn.CrossEntropyLoss()
            classifier_optimizer = torch.optim.Adam(classifier.parameters(), lr=fnn_lr)

            classifier.train()
            for epoch in range(num_classifier_epochs):
                running_loss = 0.0
                for features, labels in train_feature_loader:
                    features = features.to(device)
                    labels = labels.to(device)
                    
                    # Forward pass through classifier
                    outputs = classifier(features)
                    loss = criterion(outputs, labels)
                    
                    # Backpropagation and optimization
                    classifier_optimizer.zero_grad()
                    loss.backward()
                    classifier_optimizer.step()
                    
                    running_loss += loss.item()
                avg_loss = running_loss / len(train_feature_loader)
                print(f"Classifier Epoch {epoch+1}: loss = {avg_loss:.4f}")

        # Evaluate the classifier on test data.
        # Here we extract features from the RBM for each test image.
        if CLASSIFIER != 'LogisticRegression':
            classifier.eval()
            correct = 0
            total = 0
        features_list = []
        labels_list = []
        with torch.no_grad():
            for images, labels in test_loader:
                v = images.view(-1, 784).to(device)
                # Extract hidden activations; you can use either h_prob or h_sample.
                h_prob, _ = rbm.sample_h(v)
                if CLASSIFIER == 'LogisticRegression':
                    features_list.append(h_prob.cpu().detach().numpy())
                    labels_list.append(labels.numpy())
                else:
                    outputs = classifier(h_prob)
                    _, predicted = torch.max(outputs.data, 1)
                    total += labels.size(0)
                    correct += (predicted.cpu() == labels).sum().item()

        if CLASSIFIER == 'LogisticRegression':
            test_features = np.concatenate(features_list)
            test_labels = np.concatenate(labels_list)
            predictions = classifier.predict(test_features)
            accuracy = accuracy_score(test_labels, predictions) * 100
        
            macro_f1 = f1_score(test_labels, predictions, average="macro") 
        
        else:
            accuracy = 100 * correct / total
        
            all_preds = [] 
            all_labels = [] 
            classifier.eval()
            with torch.no_grad():
                for images, labels in test_loader:
                    v = images.view(-1, 784).to(device)
                    h_prob, _ = rbm.sample_h(v)
                    outputs = classifier(h_prob)
                    _, predicted = torch.max(outputs.data, 1)
                    all_preds.extend(predicted.cpu().numpy()) 
                    all_labels.extend(labels.numpy()) 
        
            macro_f1 = f1_score(all_labels, all_preds, average="macro") 
        
        print(f"Test Accuracy: {accuracy:.2f}%")
        print(f"Macro F1 Score: {macro_f1:.4f}") 
        
        mlflow.log_metric("test_accuracy", accuracy)
        mlflow.log_metric("macro_f1", macro_f1) 
        mlflow.end_run()
        return float(accuracy if accuracy is not None else 0.0)

if __name__ == "__main__":
    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=50)
    print(study.best_params)
    print(study.best_value)
    print(study.best_trial)

Epoch 1: avg free-energy loss = -7.9494
Epoch 2: avg free-energy loss = -16.3580
Epoch 3: avg free-energy loss = -14.2304
Epoch 4: avg free-energy loss = -12.9966
Epoch 5: avg free-energy loss = -12.2340
Classifier Epoch 1: loss = 0.7704
Classifier Epoch 2: loss = 0.5172
Classifier Epoch 3: loss = 0.4715
Classifier Epoch 4: loss = 0.4467
Classifier Epoch 5: loss = 0.4292
Test Accuracy: 83.44%
Macro F1 Score: 0.8276
Epoch 1: avg free-energy loss = -20.6368
Epoch 2: avg free-energy loss = -20.9018
Epoch 3: avg free-energy loss = -18.6343
Epoch 4: avg free-energy loss = -17.2850
Epoch 5: avg free-energy loss = -16.3783
Classifier Epoch 1: loss = 0.8672
Classifier Epoch 2: loss = 0.5673
Classifier Epoch 3: loss = 0.5158
Classifier Epoch 4: loss = 0.4849
Classifier Epoch 5: loss = 0.4681
Test Accuracy: 81.93%
Macro F1 Score: 0.8173
Epoch 1: avg free-energy loss = 296.3109
Epoch 2: avg free-energy loss = 91.2839
Epoch 3: avg free-energy loss = 62.4585
Epoch 4: avg free-energy loss = 46.8922
Epoch 5: avg free-energy loss = 39.4544
Classifier Epoch 1: loss = 0.6366
Classifier Epoch 2: loss = 0.4564
Classifier Epoch 3: loss = 0.4169
Classifier Epoch 4: loss = 0.3942
Classifier Epoch 5: loss = 0.3887
Test Accuracy: 85.08%
Macro F1 Score: 0.8486
Epoch 1: avg free-energy loss = 54.0006
Epoch 2: avg free-energy loss = 23.1856
Epoch 3: avg free-energy loss = 18.4842
Epoch 4: avg free-energy loss = 17.9494
Epoch 5: avg free-energy loss = 15.1389
Classifier Epoch 1: loss = 0.5456
Classifier Epoch 2: loss = 0.4388
Classifier Epoch 3: loss = 0.3897
Classifier Epoch 4: loss = 0.3624
Classifier Epoch 5: loss = 0.3472
Test Accuracy: 84.31%
Macro F1 Score: 0.8381
Epoch 1: avg free-energy loss = 226.4909
Epoch 2: avg free-energy loss = 67.9030
Epoch 3: avg free-energy loss = 46.5347
Epoch 4: avg free-energy loss = 36.2489
Epoch 5: avg free-energy loss = 31.9510
Classifier Epoch 1: loss = 0.6282
Classifier Epoch 2: loss = 0.4524
Classifier Epoch 3: loss = 0.4191
Classifier Epoch 4: loss = 0.4028
Classifier Epoch 5: loss = 0.3858
Test Accuracy: 84.74%
Macro F1 Score: 0.8392
Epoch 1: avg free-energy loss = 164.7099
Epoch 2: avg free-energy loss = 55.8821
Epoch 3: avg free-energy loss = 38.4960
Epoch 4: avg free-energy loss = 28.1885
Epoch 5: avg free-energy loss = 29.2875
Classifier Epoch 1: loss = 0.5411
Classifier Epoch 2: loss = 0.4347
Classifier Epoch 3: loss = 0.4121
Classifier Epoch 4: loss = 0.3994
Classifier Epoch 5: loss = 0.3632
Test Accuracy: 83.91%
Macro F1 Score: 0.8380
Epoch 1: avg free-energy loss = 53.6455
Epoch 2: avg free-energy loss = 11.5056
Epoch 3: avg free-energy loss = 7.1479
Epoch 4: avg free-energy loss = 4.5464
Epoch 5: avg free-energy loss = 3.2979
Classifier Epoch 1: loss = 0.8040
Classifier Epoch 2: loss = 0.5026
Classifier Epoch 3: loss = 0.4601
Classifier Epoch 4: loss = 0.4359
Classifier Epoch 5: loss = 0.4193
Test Accuracy: 83.97%
Macro F1 Score: 0.8386
Epoch 1: avg free-energy loss = 3.3027
Epoch 2: avg free-energy loss = -0.9166
Epoch 3: avg free-energy loss = -0.8084
Epoch 4: avg free-energy loss = -0.7366
Epoch 5: avg free-energy loss = -0.6085
Classifier Epoch 1: loss = 0.5380
Classifier Epoch 2: loss = 0.4178
Classifier Epoch 3: loss = 0.3840
Classifier Epoch 4: loss = 0.3659
Classifier Epoch 5: loss = 0.3454
Test Accuracy: 86.44%
Macro F1 Score: 0.8637
Epoch 1: avg free-energy loss = 196.7385
Epoch 2: avg free-energy loss = 59.4501
Epoch 3: avg free-energy loss = 39.2503
Epoch 4: avg free-energy loss = 30.3215
Epoch 5: avg free-energy loss = 24.9154
Classifier Epoch 1: loss = 0.5843
Classifier Epoch 2: loss = 0.4252
Classifier Epoch 3: loss = 0.4012
Classifier Epoch 4: loss = 0.3675
Classifier Epoch 5: loss = 0.3511
Test Accuracy: 85.95%
Macro F1 Score: 0.8605
Epoch 1: avg free-energy loss = 41.2625
Epoch 2: avg free-energy loss = 11.5472
Epoch 3: avg free-energy loss = 8.1285
Epoch 4: avg free-energy loss = 6.4034
Epoch 5: avg free-energy loss = 5.3511
Classifier Epoch 1: loss = 0.5370
Classifier Epoch 2: loss = 0.3989
Classifier Epoch 3: loss = 0.3668
Classifier Epoch 4: loss = 0.3484
Classifier Epoch 5: loss = 0.3259
Test Accuracy: 86.43%
Macro F1 Score: 0.8634
Epoch 1: avg free-energy loss = 75.1814
Epoch 2: avg free-energy loss = 20.2457
Epoch 3: avg free-energy loss = 14.8070
Epoch 4: avg free-energy loss = 12.2811
Epoch 5: avg free-energy loss = 10.7887
Classifier Epoch 1: loss = 0.5874
Classifier Epoch 2: loss = 0.4288
Classifier Epoch 3: loss = 0.3944
Classifier Epoch 4: loss = 0.3689
Classifier Epoch 5: loss = 0.3533
Test Accuracy: 86.48%
Macro F1 Score: 0.8623
Epoch 1: avg free-energy loss = 84.7989
Epoch 2: avg free-energy loss = 22.2254
Epoch 3: avg free-energy loss = 15.9761
Epoch 4: avg free-energy loss = 13.1407
Epoch 5: avg free-energy loss = 11.4755
Classifier Epoch 1: loss = 0.5873
Classifier Epoch 2: loss = 0.4339
Classifier Epoch 3: loss = 0.3899
Classifier Epoch 4: loss = 0.3677
Classifier Epoch 5: loss = 0.3529
Test Accuracy: 86.77%
Macro F1 Score: 0.8651
Epoch 1: avg free-energy loss = 92.5530
Epoch 2: avg free-energy loss = 25.0804
Epoch 3: avg free-energy loss = 18.2475
Epoch 4: avg free-energy loss = 14.6714
Epoch 5: avg free-energy loss = 12.6846
Classifier Epoch 1: loss = 0.6079
Classifier Epoch 2: loss = 0.4299
Classifier Epoch 3: loss = 0.4033
Classifier Epoch 4: loss = 0.3714
Classifier Epoch 5: loss = 0.3539
Test Accuracy: 86.08%
Macro F1 Score: 0.8603
Epoch 1: avg free-energy loss = 65.8722
Epoch 2: avg free-energy loss = 18.0436
Epoch 3: avg free-energy loss = 13.3529
Epoch 4: avg free-energy loss = 10.9941
Epoch 5: avg free-energy loss = 9.6961
Classifier Epoch 1: loss = 0.6006
Classifier Epoch 2: loss = 0.4269
Classifier Epoch 3: loss = 0.3847
Classifier Epoch 4: loss = 0.3657
Classifier Epoch 5: loss = 0.3514
Test Accuracy: 85.65%
Macro F1 Score: 0.8531
Epoch 1: avg free-energy loss = 176.3672
Epoch 2: avg free-energy loss = 48.0087
Epoch 3: avg free-energy loss = 31.4469
Epoch 4: avg free-energy loss = 24.4371
Epoch 5: avg free-energy loss = 20.4467
Classifier Epoch 1: loss = 0.6424
Classifier Epoch 2: loss = 0.4450
Classifier Epoch 3: loss = 0.4096
Classifier Epoch 4: loss = 0.3909
Classifier Epoch 5: loss = 0.3690
Test Accuracy: 85.61%
Macro F1 Score: 0.8556
Epoch 1: avg free-energy loss = 68.3595
Epoch 2: avg free-energy loss = 18.6610
Epoch 3: avg free-energy loss = 13.3781
Epoch 4: avg free-energy loss = 11.2155
Epoch 5: avg free-energy loss = 9.7690
Classifier Epoch 1: loss = 0.6141
Classifier Epoch 2: loss = 0.4468
Classifier Epoch 3: loss = 0.4112
Classifier Epoch 4: loss = 0.3930
Classifier Epoch 5: loss = 0.3782
Test Accuracy: 85.45%
Macro F1 Score: 0.8489
Epoch 1: avg free-energy loss = 103.7954
Epoch 2: avg free-energy loss = 28.6273
Epoch 3: avg free-energy loss = 20.5303
Epoch 4: avg free-energy loss = 16.4765
Epoch 5: avg free-energy loss = 14.3530
Classifier Epoch 1: loss = 0.5685
Classifier Epoch 2: loss = 0.4234
Classifier Epoch 3: loss = 0.3888
Classifier Epoch 4: loss = 0.3640
Classifier Epoch 5: loss = 0.3481
Test Accuracy: 85.80%
Macro F1 Score: 0.8587
Epoch 1: avg free-energy loss = 96.9961
Epoch 2: avg free-energy loss = 25.8510
Epoch 3: avg free-energy loss = 18.9783
Epoch 4: avg free-energy loss = 16.2600
Epoch 5: avg free-energy loss = 14.3283
Classifier Epoch 1: loss = 0.6185
Classifier Epoch 2: loss = 0.4455
Classifier Epoch 3: loss = 0.4120
Classifier Epoch 4: loss = 0.3797
Classifier Epoch 5: loss = 0.3625
Test Accuracy: 84.43%
Macro F1 Score: 0.8430
Epoch 1: avg free-energy loss = 59.5948
Epoch 2: avg free-energy loss = 14.7077
Epoch 3: avg free-energy loss = 10.9849
Epoch 4: avg free-energy loss = 9.0216
Epoch 5: avg free-energy loss = 7.8285
Classifier Epoch 1: loss = 0.6529
Classifier Epoch 2: loss = 0.4630
Classifier Epoch 3: loss = 0.4274
Classifier Epoch 4: loss = 0.4015
Classifier Epoch 5: loss = 0.3811
Test Accuracy: 85.17%
Macro F1 Score: 0.8474
Epoch 1: avg free-energy loss = 143.3618
Epoch 2: avg free-energy loss = 39.3630
Epoch 3: avg free-energy loss = 25.9254
Epoch 4: avg free-energy loss = 20.3991
Epoch 5: avg free-energy loss = 16.9682
Classifier Epoch 1: loss = 0.6253
Classifier Epoch 2: loss = 0.4441
Classifier Epoch 3: loss = 0.4031
Classifier Epoch 4: loss = 0.3860
Classifier Epoch 5: loss = 0.3628
Test Accuracy: 85.11%
Macro F1 Score: 0.8536
Epoch 1: avg free-energy loss = 123.2998
Epoch 2: avg free-energy loss = 28.8367
Epoch 3: avg free-energy loss = 21.0347
Epoch 4: avg free-energy loss = 17.3812
Epoch 5: avg free-energy loss = 15.1545
Classifier Epoch 1: loss = 0.6562
Classifier Epoch 2: loss = 0.4371
Classifier Epoch 3: loss = 0.4092
Classifier Epoch 4: loss = 0.3815
Classifier Epoch 5: loss = 0.3629
Test Accuracy: 85.62%
Macro F1 Score: 0.8553
Epoch 1: avg free-energy loss = 5.5166
Epoch 2: avg free-energy loss = -3.1210
Epoch 3: avg free-energy loss = -2.8285
Epoch 4: avg free-energy loss = -2.7155
Epoch 5: avg free-energy loss = -2.5241
Classifier Epoch 1: loss = 0.6257
Classifier Epoch 2: loss = 0.4527
Classifier Epoch 3: loss = 0.4197
Classifier Epoch 4: loss = 0.3943
Classifier Epoch 5: loss = 0.3813
Test Accuracy: 84.57%
Macro F1 Score: 0.8453
Epoch 1: avg free-energy loss = 13.2511
Epoch 2: avg free-energy loss = 2.9139
Epoch 3: avg free-energy loss = 2.1357
Epoch 4: avg free-energy loss = 1.7060
Epoch 5: avg free-energy loss = 1.4679
Classifier Epoch 1: loss = 0.5659
Classifier Epoch 2: loss = 0.4327
Classifier Epoch 3: loss = 0.3950
Classifier Epoch 4: loss = 0.3760
Classifier Epoch 5: loss = 0.3569
Test Accuracy: 86.06%
Macro F1 Score: 0.8607
Epoch 1: avg free-energy loss = 46.4948
Epoch 2: avg free-energy loss = 16.7855
Epoch 3: avg free-energy loss = 13.4336
Epoch 4: avg free-energy loss = 11.5313
Epoch 5: avg free-energy loss = 10.2795
Classifier Epoch 1: loss = 0.5208
Classifier Epoch 2: loss = 0.3985
Classifier Epoch 3: loss = 0.3659
Classifier Epoch 4: loss = 0.3468
Classifier Epoch 5: loss = 0.3288
Test Accuracy: 84.70%
Macro F1 Score: 0.8453
Epoch 1: avg free-energy loss = 42.8889
Epoch 2: avg free-energy loss = 11.5159
Epoch 3: avg free-energy loss = 8.3389
Epoch 4: avg free-energy loss = 6.9427
Epoch 5: avg free-energy loss = 5.9641
Classifier Epoch 1: loss = 0.5889
Classifier Epoch 2: loss = 0.4289
Classifier Epoch 3: loss = 0.3906
Classifier Epoch 4: loss = 0.3648
Classifier Epoch 5: loss = 0.3476
Test Accuracy: 86.34%
Macro F1 Score: 0.8627
Epoch 1: avg free-energy loss = 11.6034
Epoch 2: avg free-energy loss = -2.5283
Epoch 3: avg free-energy loss = -2.4783
Epoch 4: avg free-energy loss = -2.5822
Epoch 5: avg free-energy loss = -2.6208
Classifier Epoch 1: loss = 0.6669
Classifier Epoch 2: loss = 0.4696
Classifier Epoch 3: loss = 0.4285
Classifier Epoch 4: loss = 0.4026
Classifier Epoch 5: loss = 0.3877
Test Accuracy: 84.96%
Macro F1 Score: 0.8497
Epoch 1: avg free-energy loss = 88.5327
Epoch 2: avg free-energy loss = 24.7575
Epoch 3: avg free-energy loss = 17.1411
Epoch 4: avg free-energy loss = 13.7513
Epoch 5: avg free-energy loss = 11.7448
Classifier Epoch 1: loss = 0.5639
Classifier Epoch 2: loss = 0.4209
Classifier Epoch 3: loss = 0.3793
Classifier Epoch 4: loss = 0.3687
Classifier Epoch 5: loss = 0.3475
Test Accuracy: 86.94%
Macro F1 Score: 0.8699
Epoch 1: avg free-energy loss = 137.5824
Epoch 2: avg free-energy loss = 40.1476
Epoch 3: avg free-energy loss = 27.0336
Epoch 4: avg free-energy loss = 20.9861
Epoch 5: avg free-energy loss = 17.9831
Classifier Epoch 1: loss = 0.5684
Classifier Epoch 2: loss = 0.4169
Classifier Epoch 3: loss = 0.3803
Classifier Epoch 4: loss = 0.3669
Classifier Epoch 5: loss = 0.3476
Test Accuracy: 85.63%
Macro F1 Score: 0.8532
Epoch 1: avg free-energy loss = 105.2765
Epoch 2: avg free-energy loss = 28.2761
Epoch 3: avg free-energy loss = 19.5001
Epoch 4: avg free-energy loss = 15.3584
Epoch 5: avg free-energy loss = 13.1081
Classifier Epoch 1: loss = 0.5846
Classifier Epoch 2: loss = 0.4368
Classifier Epoch 3: loss = 0.4030
Classifier Epoch 4: loss = 0.3809
Classifier Epoch 5: loss = 0.3601
Test Accuracy: 86.25%
Macro F1 Score: 0.8620
Epoch 1: avg free-energy loss = 122.1296
Epoch 2: avg free-energy loss = 33.2799
Epoch 3: avg free-energy loss = 21.3056
Epoch 4: avg free-energy loss = 16.6767
Epoch 5: avg free-energy loss = 13.8188
Classifier Epoch 1: loss = 0.6627
Classifier Epoch 2: loss = 0.4575
Classifier Epoch 3: loss = 0.4158
Classifier Epoch 4: loss = 0.3925
Classifier Epoch 5: loss = 0.3698
Test Accuracy: 84.95%
Macro F1 Score: 0.8465
Epoch 1: avg free-energy loss = 170.8301
Epoch 2: avg free-energy loss = 50.4010
Epoch 3: avg free-energy loss = 34.4344
Epoch 4: avg free-energy loss = 27.1837
Epoch 5: avg free-energy loss = 22.9653
Classifier Epoch 1: loss = 0.5939
Classifier Epoch 2: loss = 0.4194
Classifier Epoch 3: loss = 0.3862
Classifier Epoch 4: loss = 0.3617
Classifier Epoch 5: loss = 0.3449
Test Accuracy: 86.08%
Macro F1 Score: 0.8597
Epoch 1: avg free-energy loss = -14.0955
Epoch 2: avg free-energy loss = -14.3351
Epoch 3: avg free-energy loss = -12.4782
Epoch 4: avg free-energy loss = -11.4570
Epoch 5: avg free-energy loss = -10.5970
Classifier Epoch 1: loss = 0.6255
Classifier Epoch 2: loss = 0.4611
Classifier Epoch 3: loss = 0.4261
Classifier Epoch 4: loss = 0.4080
Classifier Epoch 5: loss = 0.3885
Test Accuracy: 84.26%
Macro F1 Score: 0.8362
Epoch 1: avg free-energy loss = 44.0412
Epoch 2: avg free-energy loss = 7.4174
Epoch 3: avg free-energy loss = 5.9695
Epoch 4: avg free-energy loss = 4.5568
Epoch 5: avg free-energy loss = 3.8936
Classifier Epoch 1: loss = 0.6950
Classifier Epoch 2: loss = 0.4809
Classifier Epoch 3: loss = 0.4323
Classifier Epoch 4: loss = 0.4102
Classifier Epoch 5: loss = 0.3911
Test Accuracy: 85.42%
Macro F1 Score: 0.8530
Epoch 1: avg free-energy loss = 36.3599
Epoch 2: avg free-energy loss = 13.5479
Epoch 3: avg free-energy loss = 10.6614
Epoch 4: avg free-energy loss = 9.0335
Epoch 5: avg free-energy loss = 8.1207
Classifier Epoch 1: loss = 0.5690
Classifier Epoch 2: loss = 0.4272
Classifier Epoch 3: loss = 0.3881
Classifier Epoch 4: loss = 0.3619
Classifier Epoch 5: loss = 0.3392
Test Accuracy: 86.10%
Macro F1 Score: 0.8599
Epoch 1: avg free-energy loss = 149.3008
Epoch 2: avg free-energy loss = 41.9853
Epoch 3: avg free-energy loss = 27.6949
Epoch 4: avg free-energy loss = 21.0732
Epoch 5: avg free-energy loss = 17.2345
Classifier Epoch 1: loss = 0.5915
Classifier Epoch 2: loss = 0.4293
Classifier Epoch 3: loss = 0.3952
Classifier Epoch 4: loss = 0.3715
Classifier Epoch 5: loss = 0.3585
Test Accuracy: 85.87%
Macro F1 Score: 0.8576
Epoch 1: avg free-energy loss = -4.5546
Epoch 2: avg free-energy loss = -7.6506
Epoch 3: avg free-energy loss = -6.7102
Epoch 4: avg free-energy loss = -6.0152
Epoch 5: avg free-energy loss = -5.5372
Classifier Epoch 1: loss = 0.5983
Classifier Epoch 2: loss = 0.4411
Classifier Epoch 3: loss = 0.4069
Classifier Epoch 4: loss = 0.3867
Classifier Epoch 5: loss = 0.3708
Test Accuracy: 85.06%
Macro F1 Score: 0.8522
Epoch 1: avg free-energy loss = 184.0355
Epoch 2: avg free-energy loss = 35.2832
Epoch 3: avg free-energy loss = 23.2024
Epoch 4: avg free-energy loss = 18.6540
Epoch 5: avg free-energy loss = 15.7964
Classifier Epoch 1: loss = 0.6869
Classifier Epoch 2: loss = 0.4629
Classifier Epoch 3: loss = 0.4154
Classifier Epoch 4: loss = 0.3942
Classifier Epoch 5: loss = 0.3809
Test Accuracy: 85.19%
Macro F1 Score: 0.8506
Epoch 1: avg free-energy loss = 87.4798
Epoch 2: avg free-energy loss = 29.4220
Epoch 3: avg free-energy loss = 22.3986
Epoch 4: avg free-energy loss = 18.7514
Epoch 5: avg free-energy loss = 16.6470
Classifier Epoch 1: loss = 0.5979
Classifier Epoch 2: loss = 0.4341
Classifier Epoch 3: loss = 0.4029
Classifier Epoch 4: loss = 0.3807
Classifier Epoch 5: loss = 0.3651
Test Accuracy: 85.73%
Macro F1 Score: 0.8572
Epoch 1: avg free-energy loss = 19.5072
Epoch 2: avg free-energy loss = 7.3500
Epoch 3: avg free-energy loss = 5.8339
Epoch 4: avg free-energy loss = 5.0947
Epoch 5: avg free-energy loss = 4.5508
Classifier Epoch 1: loss = 0.5519
Classifier Epoch 2: loss = 0.4207
Classifier Epoch 3: loss = 0.3848
Classifier Epoch 4: loss = 0.3666
Classifier Epoch 5: loss = 0.3462
Test Accuracy: 86.52%
Macro F1 Score: 0.8638
Epoch 1: avg free-energy loss = 23.2072
Epoch 2: avg free-energy loss = 8.1844
Epoch 3: avg free-energy loss = 6.4948
Epoch 4: avg free-energy loss = 5.6222
Epoch 5: avg free-energy loss = 5.0531
Classifier Epoch 1: loss = 0.5542
Classifier Epoch 2: loss = 0.4212
Classifier Epoch 3: loss = 0.3911
Classifier Epoch 4: loss = 0.3580
Classifier Epoch 5: loss = 0.3487
Test Accuracy: 86.07%
Macro F1 Score: 0.8559
Epoch 1: avg free-energy loss = 80.4854
Epoch 2: avg free-energy loss = 22.8208
Epoch 3: avg free-energy loss = 16.2181
Epoch 4: avg free-energy loss = 13.2542
Epoch 5: avg free-energy loss = 11.2404
Classifier Epoch 1: loss = 0.5416
Classifier Epoch 2: loss = 0.4123
Classifier Epoch 3: loss = 0.3754
Classifier Epoch 4: loss = 0.3519
Classifier Epoch 5: loss = 0.3364
Test Accuracy: 86.43%
Macro F1 Score: 0.8627
Epoch 1: avg free-energy loss = -15.3423
Epoch 2: avg free-energy loss = -12.9328
Epoch 3: avg free-energy loss = -11.2064
Epoch 4: avg free-energy loss = -9.7787
Epoch 5: avg free-energy loss = -8.4751
Classifier Epoch 1: loss = 0.5575
Classifier Epoch 2: loss = 0.4289
Classifier Epoch 3: loss = 0.3926
Classifier Epoch 4: loss = 0.3761
Classifier Epoch 5: loss = 0.3573
Test Accuracy: 86.07%
Macro F1 Score: 0.8582
Epoch 1: avg free-energy loss = 38.7520
Epoch 2: avg free-energy loss = 13.7721
Epoch 3: avg free-energy loss = 10.8032
Epoch 4: avg free-energy loss = 9.3792
Epoch 5: avg free-energy loss = 8.3153
Classifier Epoch 1: loss = 0.5497
Classifier Epoch 2: loss = 0.4127
Classifier Epoch 3: loss = 0.3809
Classifier Epoch 4: loss = 0.3618
Classifier Epoch 5: loss = 0.3448
Test Accuracy: 85.11%
Macro F1 Score: 0.8481
Epoch 1: avg free-energy loss = 25.0917
Epoch 2: avg free-energy loss = 5.5691
Epoch 3: avg free-energy loss = 4.1193
Epoch 4: avg free-energy loss = 3.3557
Epoch 5: avg free-energy loss = 2.8216
Classifier Epoch 1: loss = 0.6045
Classifier Epoch 2: loss = 0.4459
Classifier Epoch 3: loss = 0.4070
Classifier Epoch 4: loss = 0.3874
Classifier Epoch 5: loss = 0.3706
Test Accuracy: 84.98%
Macro F1 Score: 0.8463
Epoch 1: avg free-energy loss = 11.7029
Epoch 2: avg free-energy loss = 0.9135
Epoch 3: avg free-energy loss = 0.1311
Epoch 4: avg free-energy loss = -0.0756
Epoch 5: avg free-energy loss = 0.0058
Classifier Epoch 1: loss = 1.0872
Classifier Epoch 2: loss = 0.5859
Classifier Epoch 3: loss = 0.5112
Classifier Epoch 4: loss = 0.4759
Classifier Epoch 5: loss = 0.4545
Test Accuracy: 82.76%
Macro F1 Score: 0.8259
Epoch 1: avg free-energy loss = 75.8773
Epoch 2: avg free-energy loss = 21.8856
Epoch 3: avg free-energy loss = 16.5895
Epoch 4: avg free-energy loss = 13.9699
Epoch 5: avg free-energy loss = 12.3491
Classifier Epoch 1: loss = 0.5630
Classifier Epoch 2: loss = 0.4183
Classifier Epoch 3: loss = 0.3716
Classifier Epoch 4: loss = 0.3596
Classifier Epoch 5: loss = 0.3411
Test Accuracy: 84.12%
Macro F1 Score: 0.8365
Epoch 1: avg free-energy loss = 54.9174
Epoch 2: avg free-energy loss = 14.4931
Epoch 3: avg free-energy loss = 10.1470
Epoch 4: avg free-energy loss = 8.2150
Epoch 5: avg free-energy loss = 7.0697
Classifier Epoch 1: loss = 0.6096
Classifier Epoch 2: loss = 0.4435
Classifier Epoch 3: loss = 0.4143
Classifier Epoch 4: loss = 0.3862
Classifier Epoch 5: loss = 0.3648
Test Accuracy: 85.29%
Macro F1 Score: 0.8505
Epoch 1: avg free-energy loss = 1.8340
Epoch 2: avg free-energy loss = -5.3326
Epoch 3: avg free-energy loss = -5.3469
Epoch 4: avg free-energy loss = -5.1137
Epoch 5: avg free-energy loss = -4.7659
Classifier Epoch 1: loss = 0.6124
Classifier Epoch 2: loss = 0.4476
Classifier Epoch 3: loss = 0.4119
Classifier Epoch 4: loss = 0.3928
Classifier Epoch 5: loss = 0.3711
Test Accuracy: 85.16%
Macro F1 Score: 0.8503
Epoch 1: avg free-energy loss = 20.3825
Epoch 2: avg free-energy loss = 8.6612
Epoch 3: avg free-energy loss = 7.0033
Epoch 4: avg free-energy loss = 5.9337
Epoch 5: avg free-energy loss = 5.4066
Classifier Epoch 1: loss = 0.5380
Classifier Epoch 2: loss = 0.4163
Classifier Epoch 3: loss = 0.3821
Classifier Epoch 4: loss = 0.3588
Classifier Epoch 5: loss = 0.3448
Test Accuracy: 86.85%
Macro F1 Score: 0.8680
Epoch 1: avg free-energy loss = 31.5311
Epoch 2: avg free-energy loss = 7.5844
Epoch 3: avg free-energy loss = 6.4449
Epoch 4: avg free-energy loss = 5.4340
Epoch 5: avg free-energy loss = 5.0087
Classifier Epoch 1: loss = 0.6789
Classifier Epoch 2: loss = 0.4783
Classifier Epoch 3: loss = 0.4428
Classifier Epoch 4: loss = 0.4150
Classifier Epoch 5: loss = 0.3989
Test Accuracy: 84.94%
Macro F1 Score: 0.8476
{'num_rbm_epochs': 5, 'batch_size': 420, 'rbm_lr': 0.0803892046509745, 'rbm_hidden': 3381, 'fnn_hidden': 256, 'fnn_lr': 0.0021046741128304017, 'num_classifier_epochs': 5}
86.94
FrozenTrial(number=26, state=1, values=[86.94], datetime_start=datetime.datetime(2025, 3, 28, 16, 44, 14, 213482), datetime_complete=datetime.datetime(2025, 3, 28, 16, 44, 42, 775743), params={'num_rbm_epochs': 5, 'batch_size': 420, 'rbm_lr': 0.0803892046509745, 'rbm_hidden': 3381, 'fnn_hidden': 256, 'fnn_lr': 0.0021046741128304017, 'num_classifier_epochs': 5}, user_attrs={}, system_attrs={}, intermediate_values={}, distributions={'num_rbm_epochs': IntDistribution(high=5, log=False, low=5, step=1), 'batch_size': IntDistribution(high=1024, log=False, low=192, step=1), 'rbm_lr': FloatDistribution(high=0.1, log=False, low=0.05, step=None), 'rbm_hidden': IntDistribution(high=8192, log=False, low=384, step=1), 'fnn_hidden': IntDistribution(high=384, log=False, low=192, step=1), 'fnn_lr': FloatDistribution(high=0.0025, log=False, low=0.0001, step=None), 'num_classifier_epochs': IntDistribution(high=5, log=False, low=5, step=1)}, trial_id=26, value=None)

[I 2025-03-28 16:31:21,441] A new study created in memory with name: no-name-7e800a9f-7085-4533-8667-8eca12c97736
[I 2025-03-28 16:31:39,948] Trial 0 finished with value: 83.44 and parameters: {'num_rbm_epochs': 5, 'batch_size': 978, 'rbm_lr': 0.07685976863360014, 'rbm_hidden': 780, 'fnn_hidden': 237, 'fnn_lr': 0.0019042488508073641, 'num_classifier_epochs': 5}. Best is trial 0 with value: 83.44.
[I 2025-03-28 16:31:58,494] Trial 1 finished with value: 81.93 and parameters: {'num_rbm_epochs': 5, 'batch_size': 841, 'rbm_lr': 0.056149697557266114, 'rbm_hidden': 462, 'fnn_hidden': 287, 'fnn_lr': 0.0010669637567547751, 'num_classifier_epochs': 5}. Best is trial 0 with value: 83.44.
[I 2025-03-28 16:32:34,604] Trial 2 finished with value: 85.08 and parameters: {'num_rbm_epochs': 5, 'batch_size': 658, 'rbm_lr': 0.08392305096214814, 'rbm_hidden': 6552, 'fnn_hidden': 267, 'fnn_lr': 0.0007830687958329613, 'num_classifier_epochs': 5}. Best is trial 2 with value: 85.08.
[I 2025-03-28 16:33:09,570] Trial 3 finished with value: 84.31 and parameters: {'num_rbm_epochs': 5, 'batch_size': 262, 'rbm_lr': 0.05171459844512646, 'rbm_hidden': 3715, 'fnn_hidden': 335, 'fnn_lr': 0.0019490479896028376, 'num_classifier_epochs': 5}. Best is trial 2 with value: 85.08.
[I 2025-03-28 16:33:57,464] Trial 4 finished with value: 84.74 and parameters: {'num_rbm_epochs': 5, 'batch_size': 303, 'rbm_lr': 0.07736123992711064, 'rbm_hidden': 8054, 'fnn_hidden': 245, 'fnn_lr': 0.00031652715400081805, 'num_classifier_epochs': 5}. Best is trial 2 with value: 85.08.
[I 2025-03-28 16:34:44,742] Trial 5 finished with value: 83.91 and parameters: {'num_rbm_epochs': 5, 'batch_size': 229, 'rbm_lr': 0.08512539443059712, 'rbm_hidden': 6476, 'fnn_hidden': 339, 'fnn_lr': 0.0010042356903125973, 'num_classifier_epochs': 5}. Best is trial 2 with value: 85.08.
[I 2025-03-28 16:35:05,006] Trial 6 finished with value: 83.97 and parameters: {'num_rbm_epochs': 5, 'batch_size': 993, 'rbm_lr': 0.09023022044286942, 'rbm_hidden': 1853, 'fnn_hidden': 379, 'fnn_lr': 0.0006805296083867429, 'num_classifier_epochs': 5}. Best is trial 2 with value: 85.08.
[I 2025-03-28 16:35:31,047] Trial 7 finished with value: 86.44 and parameters: {'num_rbm_epochs': 5, 'batch_size': 228, 'rbm_lr': 0.06038847577916692, 'rbm_hidden': 1324, 'fnn_hidden': 245, 'fnn_lr': 0.001829540306247736, 'num_classifier_epochs': 5}. Best is trial 7 with value: 86.44.
[I 2025-03-28 16:36:04,574] Trial 8 finished with value: 85.95 and parameters: {'num_rbm_epochs': 5, 'batch_size': 484, 'rbm_lr': 0.08565557594087289, 'rbm_hidden': 5310, 'fnn_hidden': 366, 'fnn_lr': 0.0012960669625454216, 'num_classifier_epochs': 5}. Best is trial 7 with value: 86.44.
[I 2025-03-28 16:36:33,760] Trial 9 finished with value: 86.43 and parameters: {'num_rbm_epochs': 5, 'batch_size': 244, 'rbm_lr': 0.09539238017567672, 'rbm_hidden': 2437, 'fnn_hidden': 316, 'fnn_lr': 0.001126868049423012, 'num_classifier_epochs': 5}. Best is trial 7 with value: 86.44.
[I 2025-03-28 16:37:01,682] Trial 10 finished with value: 86.48 and parameters: {'num_rbm_epochs': 5, 'batch_size': 443, 'rbm_lr': 0.06479703588450385, 'rbm_hidden': 3276, 'fnn_hidden': 202, 'fnn_lr': 0.002262416032800853, 'num_classifier_epochs': 5}. Best is trial 10 with value: 86.48.
[I 2025-03-28 16:37:30,052] Trial 11 finished with value: 86.77 and parameters: {'num_rbm_epochs': 5, 'batch_size': 449, 'rbm_lr': 0.0635280712474465, 'rbm_hidden': 3457, 'fnn_hidden': 198, 'fnn_lr': 0.00240717987522241, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:37:59,554] Trial 12 finished with value: 86.08 and parameters: {'num_rbm_epochs': 5, 'batch_size': 474, 'rbm_lr': 0.06514844007725767, 'rbm_hidden': 3681, 'fnn_hidden': 201, 'fnn_lr': 0.0024647160931277772, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:38:26,607] Trial 13 finished with value: 85.65 and parameters: {'num_rbm_epochs': 5, 'batch_size': 450, 'rbm_lr': 0.06776334272944196, 'rbm_hidden': 2971, 'fnn_hidden': 192, 'fnn_lr': 0.0024807254692986802, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:38:57,013] Trial 14 finished with value: 85.61 and parameters: {'num_rbm_epochs': 5, 'batch_size': 693, 'rbm_lr': 0.06821903319132003, 'rbm_hidden': 4834, 'fnn_hidden': 218, 'fnn_lr': 0.0021211953434633808, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:39:23,693] Trial 15 finished with value: 85.45 and parameters: {'num_rbm_epochs': 5, 'batch_size': 559, 'rbm_lr': 0.061374934806417356, 'rbm_hidden': 2961, 'fnn_hidden': 217, 'fnn_lr': 0.0016697531425210197, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:39:56,029] Trial 16 finished with value: 85.8 and parameters: {'num_rbm_epochs': 5, 'batch_size': 370, 'rbm_lr': 0.07183561912636137, 'rbm_hidden': 4051, 'fnn_hidden': 271, 'fnn_lr': 0.0015420604669476449, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:40:31,161] Trial 17 finished with value: 84.43 and parameters: {'num_rbm_epochs': 5, 'batch_size': 382, 'rbm_lr': 0.05193384554456661, 'rbm_hidden': 5232, 'fnn_hidden': 218, 'fnn_lr': 0.0021957625495489836, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:40:55,493] Trial 18 finished with value: 85.17 and parameters: {'num_rbm_epochs': 5, 'batch_size': 732, 'rbm_lr': 0.0601345265371627, 'rbm_hidden': 2468, 'fnn_hidden': 195, 'fnn_lr': 0.002215938520984163, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:41:25,629] Trial 19 finished with value: 85.11 and parameters: {'num_rbm_epochs': 5, 'batch_size': 579, 'rbm_lr': 0.07300285659006364, 'rbm_hidden': 4236, 'fnn_hidden': 261, 'fnn_lr': 0.0015739945469558448, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:42:05,863] Trial 20 finished with value: 85.62 and parameters: {'num_rbm_epochs': 5, 'batch_size': 366, 'rbm_lr': 0.05657302078966238, 'rbm_hidden': 6190, 'fnn_hidden': 302, 'fnn_lr': 0.0023177000521958156, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:42:27,541] Trial 21 finished with value: 84.57 and parameters: {'num_rbm_epochs': 5, 'batch_size': 526, 'rbm_lr': 0.06314711024848019, 'rbm_hidden': 1319, 'fnn_hidden': 240, 'fnn_lr': 0.0018888821833494105, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:42:52,038] Trial 22 finished with value: 86.06 and parameters: {'num_rbm_epochs': 5, 'batch_size': 333, 'rbm_lr': 0.05779878497371583, 'rbm_hidden': 1639, 'fnn_hidden': 227, 'fnn_lr': 0.002006485444042037, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:43:28,293] Trial 23 finished with value: 84.7 and parameters: {'num_rbm_epochs': 5, 'batch_size': 195, 'rbm_lr': 0.06758320673233065, 'rbm_hidden': 3102, 'fnn_hidden': 204, 'fnn_lr': 0.0016767186968677668, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:43:52,729] Trial 24 finished with value: 86.34 and parameters: {'num_rbm_epochs': 5, 'batch_size': 443, 'rbm_lr': 0.06359326976695494, 'rbm_hidden': 2332, 'fnn_hidden': 244, 'fnn_lr': 0.002366610247557912, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:44:14,213] Trial 25 finished with value: 84.96 and parameters: {'num_rbm_epochs': 5, 'batch_size': 626, 'rbm_lr': 0.07109375377447083, 'rbm_hidden': 1350, 'fnn_hidden': 211, 'fnn_lr': 0.00178124286827835, 'num_classifier_epochs': 5}. Best is trial 11 with value: 86.77.
[I 2025-03-28 16:44:42,775] Trial 26 finished with value: 86.94 and parameters: {'num_rbm_epochs': 5, 'batch_size': 420, 'rbm_lr': 0.0803892046509745, 'rbm_hidden': 3381, 'fnn_hidden': 256, 'fnn_lr': 0.0021046741128304017, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:45:16,326] Trial 27 finished with value: 85.63 and parameters: {'num_rbm_epochs': 5, 'batch_size': 415, 'rbm_lr': 0.0760341429394765, 'rbm_hidden': 4646, 'fnn_hidden': 277, 'fnn_lr': 0.002127170193911178, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:45:44,672] Trial 28 finished with value: 86.25 and parameters: {'num_rbm_epochs': 5, 'batch_size': 513, 'rbm_lr': 0.07819930820339267, 'rbm_hidden': 3528, 'fnn_hidden': 257, 'fnn_lr': 0.0013928438939582132, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:46:10,733] Trial 29 finished with value: 84.95 and parameters: {'num_rbm_epochs': 5, 'batch_size': 755, 'rbm_lr': 0.07942647189415598, 'rbm_hidden': 3292, 'fnn_hidden': 226, 'fnn_lr': 0.002043828790787277, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:46:51,821] Trial 30 finished with value: 86.08 and parameters: {'num_rbm_epochs': 5, 'batch_size': 321, 'rbm_lr': 0.08186871219139809, 'rbm_hidden': 5893, 'fnn_hidden': 228, 'fnn_lr': 0.002290381822237252, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:47:12,952] Trial 31 finished with value: 84.26 and parameters: {'num_rbm_epochs': 5, 'batch_size': 419, 'rbm_lr': 0.05934462152347368, 'rbm_hidden': 731, 'fnn_hidden': 257, 'fnn_lr': 0.001840087350007031, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:47:34,786] Trial 32 finished with value: 85.42 and parameters: {'num_rbm_epochs': 5, 'batch_size': 908, 'rbm_lr': 0.06571480741451359, 'rbm_hidden': 1878, 'fnn_hidden': 284, 'fnn_lr': 0.0022925355152995383, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:48:04,924] Trial 33 finished with value: 86.1 and parameters: {'num_rbm_epochs': 5, 'batch_size': 287, 'rbm_lr': 0.05340130938663253, 'rbm_hidden': 2701, 'fnn_hidden': 301, 'fnn_lr': 0.0024647338173498577, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:48:33,413] Trial 34 finished with value: 85.87 and parameters: {'num_rbm_epochs': 5, 'batch_size': 597, 'rbm_lr': 0.08950918746667691, 'rbm_hidden': 3917, 'fnn_hidden': 206, 'fnn_lr': 0.0020608827395373684, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:48:55,594] Trial 35 finished with value: 85.06 and parameters: {'num_rbm_epochs': 5, 'batch_size': 393, 'rbm_lr': 0.07025938242651668, 'rbm_hidden': 1017, 'fnn_hidden': 234, 'fnn_lr': 0.001912352872565176, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:49:36,383] Trial 36 finished with value: 85.19 and parameters: {'num_rbm_epochs': 5, 'batch_size': 538, 'rbm_lr': 0.055149846639407245, 'rbm_hidden': 7204, 'fnn_hidden': 248, 'fnn_lr': 0.001756580048101243, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:50:17,003] Trial 37 finished with value: 85.73 and parameters: {'num_rbm_epochs': 5, 'batch_size': 197, 'rbm_lr': 0.07478335307044473, 'rbm_hidden': 4574, 'fnn_hidden': 250, 'fnn_lr': 0.0003289509829743759, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:50:44,811] Trial 38 finished with value: 86.52 and parameters: {'num_rbm_epochs': 5, 'batch_size': 271, 'rbm_lr': 0.05051354616221276, 'rbm_hidden': 2035, 'fnn_hidden': 235, 'fnn_lr': 0.0022094590698111274, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:51:12,045] Trial 39 finished with value: 86.07 and parameters: {'num_rbm_epochs': 5, 'batch_size': 289, 'rbm_lr': 0.05092809628763094, 'rbm_hidden': 2125, 'fnn_hidden': 235, 'fnn_lr': 0.002387358167458787, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:51:42,199] Trial 40 finished with value: 86.43 and parameters: {'num_rbm_epochs': 5, 'batch_size': 339, 'rbm_lr': 0.08173771866243773, 'rbm_hidden': 3358, 'fnn_hidden': 213, 'fnn_lr': 0.002198761976074191, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:52:06,075] Trial 41 finished with value: 86.07 and parameters: {'num_rbm_epochs': 5, 'batch_size': 244, 'rbm_lr': 0.061155297945488796, 'rbm_hidden': 652, 'fnn_hidden': 270, 'fnn_lr': 0.001938790689717326, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:52:36,261] Trial 42 finished with value: 85.11 and parameters: {'num_rbm_epochs': 5, 'batch_size': 272, 'rbm_lr': 0.054546146445554416, 'rbm_hidden': 2846, 'fnn_hidden': 202, 'fnn_lr': 0.0021845015895273624, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:53:00,157] Trial 43 finished with value: 84.98 and parameters: {'num_rbm_epochs': 5, 'batch_size': 479, 'rbm_lr': 0.05722994837175465, 'rbm_hidden': 1867, 'fnn_hidden': 230, 'fnn_lr': 0.0020059505572936343, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:53:27,020] Trial 44 finished with value: 82.76 and parameters: {'num_rbm_epochs': 5, 'batch_size': 241, 'rbm_lr': 0.09772530064668687, 'rbm_hidden': 1459, 'fnn_hidden': 195, 'fnn_lr': 0.00010373162465527382, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:53:59,057] Trial 45 finished with value: 84.12 and parameters: {'num_rbm_epochs': 5, 'batch_size': 319, 'rbm_lr': 0.06365930229465193, 'rbm_hidden': 3710, 'fnn_hidden': 222, 'fnn_lr': 0.002394146804012174, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:54:26,014] Trial 46 finished with value: 85.29 and parameters: {'num_rbm_epochs': 5, 'batch_size': 454, 'rbm_lr': 0.0739325096649262, 'rbm_hidden': 2477, 'fnn_hidden': 341, 'fnn_lr': 0.0008273701764834851, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:54:47,959] Trial 47 finished with value: 85.16 and parameters: {'num_rbm_epochs': 5, 'batch_size': 498, 'rbm_lr': 0.08739263811387749, 'rbm_hidden': 1111, 'fnn_hidden': 239, 'fnn_lr': 0.0021139215380483072, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:55:18,493] Trial 48 finished with value: 86.85 and parameters: {'num_rbm_epochs': 5, 'batch_size': 218, 'rbm_lr': 0.05023961955170966, 'rbm_hidden': 2133, 'fnn_hidden': 253, 'fnn_lr': 0.0013908896355394004, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.
[I 2025-03-28 16:55:41,367] Trial 49 finished with value: 84.94 and parameters: {'num_rbm_epochs': 5, 'batch_size': 637, 'rbm_lr': 0.05202843720829133, 'rbm_hidden': 2113, 'fnn_hidden': 262, 'fnn_lr': 0.0010786691647612355, 'num_classifier_epochs': 5}. Best is trial 26 with value: 86.94.

Test Accuracy by RBM Hidden Units

Test Accuracy by FNN Hidden Units

Model	Optuna Best Trial MLflow Test Accuracy(%)	Macro F1 Score
Logistic Regression	84.71	0.846
Feed Forward Network	88.06	0.879
Convolutional Neural Network	91.29	0.913
Logistic Regression (on RBM Hidden Features)	87.14	0.871
Feed Forward Network (on RBM Hidden Features)	86.95	0.869

Conclusion

• Summarize your key findings.

CNN clearly outperforms other models. Logistic Regression, which typically performs well for binary classifications tasks, underperforms on Fashion MNIST multiclassification task. Logistic Regression is improved by using a Restricted Boltzmann Machine first to extract the hidden features from the input data prior to classification. Feed Forward Network is not improved by the use of RBM. These findings clearly show the progress in machine and deep learning and how more advanced neural networks on raw pixels can outperform models that use RBM hidden features.

• Discuss the implications of your results.

References

Akiba, Takuya, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. “Optuna: A Next-Generation Hyperparameter Optimization Framework.” In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2623–31.

Aslan, Narin, Sengul Dogan, and Gonca Ozmen Koca. 2023. “Automated Classification of Brain Diseases Using the Restricted Boltzmann Machine and the Generative Adversarial Network.” Engineering Applications of Artificial Intelligence 126: 106794.

Fiore, Ugo, Francesco Palmieri, Aniello Castiglione, and Alfredo De Santis. 2013. “Network Anomaly Detection with the Restricted Boltzmann Machine.” Neurocomputing 122: 13–23.

Fischer, Asja, and Christian Igel. 2012. “An Introduction to Restricted Boltzmann Machines.” In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 17th Iberoamerican Congress, CIARP 2012, Buenos Aires, Argentina, September 3-6, 2012. Proceedings 17, 14–36. Springer.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.

Hinton, Geoffrey. 2010. “A Practical Guide to Training Restricted Boltzmann Machines.” Momentum 9 (1): 926.

Hinton, Geoffrey E. 2002. “Training Products of Experts by Minimizing Contrastive Divergence.” Neural Computation 14 (8): 1771–1800.

Melko, Roger G, Giuseppe Carleo, Juan Carrasquilla, and J Ignacio Cirac. 2019. “Restricted Boltzmann Machines in Quantum Physics.” Nature Physics 15 (9): 887–92.

Ning, Lin, Randall Pittman, and Xipeng Shen. 2018. “LCD: A Fast Contrastive Divergence Based Algorithm for Restricted Boltzmann Machine.” Neural Networks 108: 399–410.

O’Shea, Keiron, and Ryan Nash. 2015. “An Introduction to Convolutional Neural Networks.” https://arxiv.org/abs/1511.08458.

Oh, Sangchul, Abdelkader Baggag, and Hyunchul Nha. 2020. “Entropy, Free Energy, and Work of Restricted Boltzmann Machines.” Entropy 22 (5): 538.

Peng, Chao-Ying Joanne, Kuk Lida Lee, and Gary M Ingersoll. 2002. “An Introduction to Logistic Regression Analysis and Reporting.” The Journal of Educational Research 96 (1): 3–14.

Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. 2007. “Restricted Boltzmann Machines for Collaborative Filtering.” In Proceedings of the 24th International Conference on Machine Learning, 791–98.

Sazlı, Murat H. 2006. “A Brief Review of Feed-Forward Neural Networks.” Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering 50 (01).

Smolensky, Paul et al. 1986. “Information Processing in Dynamical Systems: Foundations of Harmony Theory.”

Xiao, Han, Kashif Rasul, and Roland Vollgraf. 2017. “Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms.” August 28, 2017. https://arxiv.org/abs/cs.LG/1708.07747.

Zaharia, Matei, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, et al. 2018. “Accelerating the Machine Learning Lifecycle with MLflow.” IEEE Data Eng. Bull. 41 (4): 39–45.

Zhang, Nan, Shifei Ding, Jian Zhang, and Yu Xue. 2018. “An Overview on Restricted Boltzmann Machines.” Neurocomputing 275: 1186–99.